What my AI knowledge-system workshop for two insurance advisors got wrong
Two insurance advisors kept re-explaining themselves to ChatGPT and losing what they'd learned across old chats. I ran a workshop to give them a second brain they own. It produced real excitement but no proof it worked on the day. Here is what broke and what I changed.
Every time they opened ChatGPT, two insurance advisors started from zero. They re-typed who they were, which client they were meeting and which policies were involved. A week later, all of that was buried in a pile of old conversations. The information existed somewhere, but when a client mentioned a policy mid-meeting, they couldn't pull it up fast enough, and sometimes couldn't even remember the name to search for.
"When I'm retrieving the data, it's a little bit not so consistent. I'm not sure where it is."
What they needed: a second brain they own
What I proposed was a knowledge base they keep themselves: client notes, product summaries and meeting records in one folder of plain files that any AI assistant can read. The same approach runs a private search engine over 13,000 notes.
- No more re-typing context. The assistant reads their notes instead of asking them to explain their business again.
- It compounds. Every note they save makes the next answer better, which makes saving the next note worth it. Instead of knowledge leaking away across chats, it builds on itself.
- No vendor lock-in. The files belong to them. If ChatGPT, Claude or a better app comes along next year, they point the new tool at the same folder and keep everything they've built.
That was the promise. This is the story of the workshop where I didn't deliver it on the day, and what I changed so the next one does.
What happened
The first nine minutes were the best part. We defined the use cases, the top questions clients ask, and what "useful" would look like.
Then we started setting things up, and the session turned into live troubleshooting:
- Choosing between ChatGPT, Claude, Codex, NotebookLM and others, and working out which ones charge extra.
- Granting file permissions, then working out why some folders were readable and others weren't.
- Moving files from Google Drive and Dropbox, then discovering two separate copies of the knowledge base on the same laptop.
- Converting PDFs, and finding that some appeared in the file picker but were never actually indexed.
- Looking at private insurer portals that need a person to log in, a code sent to their phone, and sometimes a specific browser.
There were real moments of delight. When the knowledge graph started filling in, one advisor said, "My brain is growing." But when I reviewed the recording afterwards, I had to write down the uncomfortable fact:
At no point did either advisor ask the system a real work question and get a trustworthy answer. The recording ends with us still moving the correct folder into place.
Why it went wrong
Nothing was wrong with the technology. The session design was wrong.
No finish line
We never agreed on what "done" meant. With an 80 GB pile of source material, "put everything in" can never be finished.
Architecture came after the clicking
They needed one simple picture first: where files live, what syncs them, what the AI can read, and what it can change. Without it, every permission prompt felt risky, and it was.
Too many tools at once
Eleven overlapping products came up in one session. Every one of them was a decision the advisors didn't need to make on day one.
The hardest data came first
Private portals with phone-code logins and customised benefit illustrations are the worst place to start. General product summaries and their own meeting notes would have proved the value in minutes.
What I changed
One of the advisors told me directly that a preflight checklist was the thing he'd needed most. He was right, so every knowledge-system engagement now follows a stricter format:
Before
Install the tools live, bring in everything, and hope the graph looks impressive by the end.
After
Tools installed before the session. Start with 5–10 files that answer a question the client already asks every week. Finish with a retrieval test, not a demo.
- A preflight checklist, sent days before, covering accounts, subscriptions, permissions, where the files live, and which ones are sensitive.
- A 5–10 file first batch, chosen because they answer real, recurring questions, not because they're the complete archive.
- One diagram before any setup, showing where the files live, how they sync, and what the AI can read and change.
- Sensitive-data rules agreed up front: what never goes in, what stays local, and what can use a cloud model.
- Every session ends with a retrieval test. The client asks three questions from their actual week. If the answers aren't right and sourced, we're not done.
- A maintenance routine, because policies and client notes change every week. A knowledge base nobody updates is out of date within a month.
Why I'm publishing a session that went badly
Because this is what most AI adoption looks like when you look closely. The demo is exciting and the graph is beautiful, and the business value is still unproven. The failure wasn't that the advisors couldn't use AI. It was that I let the setup become the goal, instead of one question answered correctly.