AI chatbots can save you hours every week, or cost you your customers' trust
A chatbot that answers confidently and wrongly is worse than no chatbot. I built one that grades every answer before a customer sees it and only sends the ones that pass. Here's how the idea works and how to apply it.
One confident wrong answer about a refund or a price, and a customer stops believing anything your website says. That's the real risk of putting an AI chatbot in front of customers, and it's why I built one that grades every answer before a customer sees it.
The upside is real too. A chatbot can save a small business hours every week answering the same questions about prices, bookings, policies and opening hours. The trick is getting the hours without losing the trust.
Why "give it your documents" isn't enough
Most business chatbots use a method called retrieval: when a question comes in, the system searches your documents for relevant passages and hands them to the AI to write an answer. It's a good approach, and it fails in more ways than people expect:
- The search finds the wrong passage, such as last year's price list instead of this year's.
- It finds the right passage but misses the exception on the next page.
- The AI gets the right context and still adds a detail that isn't there.
- Several small errors stack up into one confidently wrong answer.
Getting the right context is necessary. It isn't sufficient.
A chatbot with a teacher behind it
So in 2025 I built a chatbot that works more like a student with a strict teacher:
Draft
The first agent writes an answer from the retrieved documents, as usual.
Grade
A second agent, which didn't write the answer, scores it out of 100 against the source material. Is every claim supported? Is anything missing? Does it answer the question actually asked?
Revise or stop
Answers scoring below 90 go back with the grader's feedback for another attempt. Only answers that pass are sent as a verified reply. If an answer can't pass, the bot says it doesn't know and hands over to a person.
I built it with n8n, an automation tool, plus custom agent logic, and tested it on a deliberately difficult subject: Chinese metaphysics, where the source material is dense and a plausible-sounding wrong answer is easy to produce.
The principle is the same one behind my multi-agent systems today. The agent that writes should never be the one that approves.
What this means in practice
For a small business, the grading step changes the maths. Without it, the chatbot's worst answers go straight to customers and you find out from complaints. With it, the worst answers become "Let me get someone to help you with that", which customers forgive easily.
A few rules worth copying:
- Set a pass mark and mean it. For customer-facing answers, I'd rather the bot hand over too often than guess.
- Make "I don't know" a feature. A clean handover to a person with the conversation attached builds more trust than a clever guess.
- Grade against the source, not the draft. The grader needs the original documents, or it will simply be persuaded by a well-written answer.
- Review the failures weekly. Questions that keep failing tell you what's missing or unclear in your own documents, which is useful even without the chatbot.
You don't need a huge test set to start. Anthropic's guide to evaluating AI agents says "20-50 simple tasks drawn from real failures is a great start". Those weekly failures are exactly where your list comes from.
The pass mark is one example of a broader rule: act fast on the answers an AI is sure about, and check the ones it isn't. I explain it in Two kinds of decisions.
Originally posted on LinkedIn
The original post, with a video walkthrough of the system.