MIT found 1 in 20 AI pilots pay off. What three small AI rollouts showed about fit

Of the enterprise AI pilots MIT's NANDA initiative studied, about 1 in 20 was producing real value, and the rest mostly failed on fit with the work, the gap forward-deployed engineers are hired to close. That is after $30–40 billion of company spending on generative AI. The July 2025 report, The GenAI Divide: State of AI in Business 2025, found that "just 5% of integrated AI pilots are extracting millions in value, while the vast majority remain stuck with no measurable P&L impact." The authors write that the divide "does not seem to be driven by model quality or regulation", and that most tools fail due to "brittle workflows, lack of contextual learning, and misalignment with day-to-day operations."

A 2006 Harvard Business Review paper helps explain why that misalignment is hard to overcome. In "Eager Sellers and Stony Buyers", John Gourville of Harvard Business School applied Daniel Kahneman and Amos Tversky's work on loss aversion to new products. His estimate: people overvalue what they already use by about three times, and the people offering something new overvalue it by about three times. That leaves a rough 9x gap between what the builder thinks a new tool is worth and what the user will give up to switch.

The three cases below are small: a workshop for two insurance advisors, an audit of a 50-person home-repair group, and user tests with four insurance advisers. They don't prove the MIT finding. They show what "misalignment with day-to-day operations" can look like up close, and they suggest a response: scope each rollout the way a founder looks for product-market fit (Marc Andreessen's term for the point where people use something without being pushed), one team at a time.

Case 1: a workshop that started from the tools

A Flareworks workshop for two insurance advisors aimed to give them one place to keep what they had learned, instead of re-explaining themselves to ChatGPT in every new chat.

The recorded session ran 79 minutes: 9 minutes of discovery and 70 minutes hands-on. During it, the advisors named 11 overlapping AI tools they already paid for or had tried, and wanted to bring in about 80 GB of old material. The room was enthusiastic. On the recording, 0 real work questions were answered.

What it showed: the session was scoped around tools and material, not around one question the advisors needed answered every week. Enthusiasm in the room was not the same as use at the desk. The workshop case study covers what was changed afterwards.

Case 2: an audit where the best answer was less automation

In an October 2026 Flareworks audit, the owner of a 50-person home-repair and appliance group had already decided that the enquiry and the quote would stay with a person: "It has to be human touch and not automated. It's deliberately slow." For her customers, a real person on the first call is part of the service.

The same business had tried a team tool and dropped it, because technicians wouldn't log in. When a new dashboard came up, her question was immediate: "Who's gonna log in daily?" Her job software had been in place for over ten years and was, in her words, "almost impossible to pull out".

What it showed: this looks like Gourville's gap in a real business. The old system had ten years of habits behind it, and the earlier tool asked technicians to take on a new daily habit, which they didn't. The audit's proposed scope went behind the counter instead: answers drawn from the company's own documents, with the customer-facing steps left as they were. These are recommendations, not results yet. The full audit is in Every AI proposal has a second spec.

Case 3: user tests that narrowed the product

At Mindreader, documented user-acceptance testing with four insurance advisers, including two rounds with the same adviser, showed where technically correct AI output was not useful in practice. Experienced advisers felt they already knew how to handle the conversation. Newer advisers had a clearer need.

What it showed: the product narrowed from "AI insight for every adviser" to guided practice for newer advisers and their team leads. That narrowing came from watching people do their real work, not from the demo. It is described in the forward-deployed AI case study.

This is the work the forward-deployed engineer role exists for, a term Palantir popularised for engineers placed inside a customer's team. It follows the same loop a founder uses to look for product-market fit: sit with the users, ask the questions they aren't asking themselves, build the smallest useful thing, and choose the interface on purpose, whether that is a tool for one analyst or something a technician can use from a phone in ten seconds. Inside a company the users are close by, so the loop can be short.

What the three cases had in common

Across the three cases, the same four questions came up:

• Who the user was: two advisors, a group of technicians, newer advisers. Not "everyone".
• What the real process was: in the advisors' heads, in ten-year-old job software, in live client conversations. None of it was written down where a system could use it, which is why writing down one process comes first.
• What should stay human: in Case 2, the first call and the quote.
• What would count as success: a real question answered, a technician finding an answer without logging into something new, an adviser practising a hard conversation.

Two common shortcuts skip those questions. Cutting staff before a system has proven itself risks losing the people who know where the unwritten steps are, the steps Case 2's owner kept in her own head. Training everyone and counting logins measures attendance, not use, which is the gap Case 1 recorded.

Grants pay for the tool, not the scoping

Singapore's support for adoption is real. The Productivity Solutions Grant and the Enterprise Development Grant reduce the cost of new tools, and in October 2026 the government set aside S$45 million to help advanced-manufacturing SMEs adopt AI.

A grant lowers the cost of trying. It doesn't do the scoping in the cases above: naming the user, writing down the process, deciding what stays human and agreeing what to check. The one-lever rule is the shortest version of that test: name the number the project moves, from what to what, and who checks it.