A 4.8 out of 5 training score isn't a product: education is not usage
Founders who teach well often assume a great product will follow. Training scores measure whether people enjoyed the room. Software lives or dies on whether they change what they do on Tuesday afternoon, and that is a different discipline.
In March and April 2023, Mindreader's sales training for AIA advisors scored between 4.83 and 4.97 out of 5, and every respondent said they would recommend it. In September 2026, a review of 250 Mindreader user accounts found that 110 had never run a single analysis in the software.
Both numbers describe the same kind of customer. The first measures whether people enjoyed learning. The second measures whether the tool became part of their week. Founders who teach well, and owners who buy AI training for their staff, often treat the first as a sign of the second. It isn't.
Four levels, and where the scores sit
In the 1950s, Donald Kirkpatrick proposed a way to evaluate training that is still the standard reference. Kirkpatrick Partners describes four levels:
Level 1: Reaction
Did people find it favourable, engaging and relevant?Level 2: Learning
Did they gain the knowledge, skills, confidence and commitment?Level 3: Behaviour
Do they actually perform the new behaviours back at work, with support and accountability?Level 4: Results
Did the business outcomes the training aimed for really happen?
Almost every workshop feedback form measures Level 1. Some measure Level 2. Very few ever reach Levels 3 and 4, because by then the trainer has gone home.
The implication for software is that it is judged almost entirely at Levels 3 and 4. Nobody renews a subscription because the onboarding was engaging. They renew because the tool became part of how they work, and something measurable got better. A product doesn't get a feedback form at the end of the day. It gets a usage log, every day, forever.
The receipts: brilliant training, a separate problem
At Mindreader, the team ran sales training for AIA advisors in March and April 2023, with Ethan Lin as lead trainer. Each run had a formal evaluation. Across two cohorts, average scores ranged from 4.83 to 4.97 out of 5:
That was real, earned, and a credit to Ethan and the team. Over its life, Mindreader's software grew to more than 3,000 users and more than 100 paying customers. But usage told a different story from the training scores:
- In September 2026, of 250 Mindreader user accounts reviewed as prospects, 175 sat inside seven financial-services firms, and 110 had never run a single analysis. Many were bulk sign-ups from 2022 that never activated.
- In January–June 2025, 438 free accounts produced 139 first analyses, 8 repeat analyses within 30 days, and no paid conversions. The free plan was removed in favour of a 14-day trial.
- In 15 independent interviews in May–June 2026, the most committed insurance-agent user had created 18 profiles in 10 months, with no closed deals attributed to the tool.
Both facts are true at once. People loved learning with us, and the tool did not become part of their week. The gap between those two numbers is the gap between Level 1 and Level 3. Mindreader is now moving away from the SaaS model toward AI services, helping teams design the workflows around the insight rather than selling another app. I wrote about why in the Mindreader case study.
I'm seeing the same pattern again with The Modern Shrine, my consumer AI product. As of August 2026 it has run 13 workshops across Malaysia, Singapore, Vietnam and Japan, with more than 100 participants. The workshops create warmth and trust. In July, two paid deals closed from older leads who came back. But workshops don't by themselves make people open the app next week. That part has to be designed.
Why the jump from course to software is hard
From the coaches and salespeople I've met, and from Mindreader, three problems come up again and again.
Distribution. If your only channel is your training, two things happen. The people you reach may not be the right users for the software, because they came to learn, not to change their tools. And training alone caps how many people you can reach. At Mindreader, the team relied heavily on training for early customers and never broke into another distribution model that let the software scale, even though software in theory has high margins and good unit economics. In a training room, trust, being human and personal relationships are taken for granted. None of that carries over when you sell pure online software.
Building. Many founders from teaching hire offshore developers to build the first version. Without technical expertise, they can't fact-check the work, so they end up overpaying, overbuilding, or accepting a bad quote without knowing it.
Attention. Most underestimate how hard it is to get anyone to notice a product. Models are now smart enough that a convincing demo can be built overnight, so more products than ever compete for the same attention. Standing out is tough.
Workshops grow a business, until they hit a ceiling
I want to be fair to teaching. Workshops, live relationships and referrals are one of the best ways to grow an early business. You learn your customers' language, build trust quickly and get paid while you learn.
The ceiling is your calendar. Past a certain point, growth needs a system that works inside the customer's day when you're not there. Building it is a different discipline from teaching:
- The room does the motivating. In a workshop, your energy and the group keep people engaged. At 4pm on a Tuesday, alone with a login screen, there's no room.
- Teaching tolerates friction. Software doesn't. A confusing step in a workshop gets a question and an answer. In an app, it gets a closed tab. When The Modern Shrine's workshops moved from ChatGPT to its own app in July 2026, day one brought login and unlock confusion that the room could absorb; it was fixed by day two.
- Understanding isn't habit. Someone can understand your method perfectly and still never change their routine to use a tool for it.
The wider research points the same way. BCG's AI at Work 2026 found that 74% of frontline staff are regular AI users, but 66% get little or no guidance on how to use the time they save. Microsoft's Work Trend Index 2026 attributed 67% of AI's reported impact to organisational factors, and only 32% to individuals. Access and enthusiasm are Level 1. Results come from how the work around the tool is set up.
The real question before you build
The real question isn't "will people like this?" Your workshop scores already answered that. It's "what will people do differently, every week, because this exists, and what will that change?"
This is most of what I do now as a fractional CTO or CPO for founders: checking whether what they plan to build matches their goals, positioning and distribution before the money goes into code, and, when something is worth building, helping them brief and check developers so they don't overpay or overbuild. I've built software that served peaks of 250,000 requests a minute at Rakuten, and I've been CTO of a SaaS company that learned this lesson with real users. Scaling the servers was the easier part. Scaling the behaviour was harder.
Sometimes the right answer is an app. Often it's a smaller system inside a tool the customer already uses, or a service with software behind it. Occasionally it's "keep teaching, and raise your prices." If you're weighing that decision, I'm happy to talk it through.