My CRM is an agent system. Evals are how I know it works.

This is an internal production system. It runs my own consultancy, Flareworks. It isn't something I built for a client, and nothing here claims a client result.

Running a small AI consultancy means a lot of small jobs that share the same information: finding companies, checking they're real, working out who to contact, writing to them and remembering to follow up. I treat the whole thing as one system with a single source of truth, and I'm the only person who can send anything.

The first version was fast and busy. What it couldn't tell me was whether it was right. This is the system, what version 1 produced, what went wrong, and how evals turned it into something I can measure.

The system

The source of truth is one Notion database, "Flareworks: Singapore Leads". Every lead is a row: who they are, which lane found them, the evidence, an ICP score out of 100 (how closely they match the customers I want), whether the email is verified, what stage they're at, and a follow-up date.

1. Find

   Each scheduled job searches one source ("lane"): people who reacted to relevant LinkedIn posts, Google reviews that show a firm losing customers before the sale, and hiring ads from small companies. The searches go through treg, a service that sells public data per lookup. Business cards from events come in through a photo in the Claude app.

2. Enrich

   Before a lead is saved, the system checks the website, phone, owner and a work email. Enrichment throws many leads out.

3. Score

   Each lead gets an ICP score, which decides who I write to first.

4. Draft

   ChatGPT reads the notes on each lead and drafts the email.

5. Approve and send

   I read every draft, then send it or don't, and set the follow-up date.

The scheduled jobs look things up and write rows. None of them has a send step, and every email goes out from my own Gmail after I've read it. The treg keys can only buy data lookups and have a spend cap of about US$20 a month, so that's the most a mistake can cost.

Version 1, by the numbers

After 25 days (as of 2 October 2026):

Leads in the CRM	424
With a verified email	211, about half
Sent	165
Marked replied	11, two of which reached a short discovery or findings stage
Time saved (my estimate)	~70 hours over 25 days, mostly in finding and checking, not writing

The time estimate is arithmetic, not a stopwatch: about 10 minutes to find, check and score a lead by hand, about 5 to write an email, and 20 minutes a day working out who's waiting on me. Against that, I spend about 45 minutes a day reading, approving and sending.

Those are activity numbers. They say the system is busy. They don't say it's right, and reading every email before it goes out only checks the writing. It doesn't check the facts the writing was built on.

What went wrong in version 1

Version 1 ran on Claude Code for research and ChatGPT for drafts. Both are semi-autonomous, so I couldn't see every step they took, only what came out. These problems turned up in the first month, mostly by hand:

The AI believed job titles. On the third LinkedIn run it rated eight people as strong leads who weren't. A "Finance Director" at a large company looked like the owner of a small one. Nobody was contacted, because I review high-scoring leads before they're saved. Enrichment now checks company size and ownership, and a lead that can't be confirmed is dropped.

Wrong websites. In one pull of 570 businesses, a hand review caught at least six matched to the wrong organisation: a renovation firm matched to a shopping mall, a medical practice to an overseas hospital network, an aviation company to a government regulator, and a design studio to a university abroad. The pipeline flagged most of them, but one wrong match went through as "ready to push" with no flag at all.

Wrong numbers in the opener. Each email opened with a market figure from my job-ad research, such as the share of small-business admin roles that involve answering enquiries. The industry figures had quietly mixed in technician roles, so the percentages were off. They were already written into 100 lead records, which I rewrote by hand.

False duplicates. The check that stops me contacting a business twice matched on names loosely. One generic business name made five unrelated businesses look like existing contacts, so real leads were silently dropped.

One thing went right: a firm scored 85, but its site returned a blank page to the scraper, so the system wouldn't write a line about the website and left a note to check by hand. A note that says "I couldn't see this" is better than a guess.

Every one of these was caught by luck or by me. None of them would have shown up as a number.

Version 2: a contract per stage

I don't control the agents' internals, so I don't evaluate them. I evaluate what each stage must deliver: given this input, what must it do, and what must it never do?

Stage	Contract
Find the business	The website belongs to the business named in the source, judged against its registered name
Find the email	The email domain matches the website, or a free-mail address is clearly tied to the business
Verify	Only verified, catch-all or risky addresses get emailed
Dedupe	One address is never contacted twice; a business reached through several people is flagged
Respect warm routes	Anyone I already talk to never gets a cold email
Personalise	Every claim about the business traces to the research record
Write	Specific, relevant and short; never "you don't need to hire"; no sensitive topics; no placeholders

Each line became a check. The suite reads a snapshot of the CRM and the sent emails, and never writes to either. It runs in three layers:

1. Rules, on everything, for free. Most of the contract needs no AI: domain matching, verification status, duplicates, word count, placeholders, follow-up spacing. They run on every lead and every email in seconds.
2. A golden set for the riskiest step. Fifty businesses with the website the pipeline found, each labelled correct or wrong by hand. Any change to the matching step is rerun against the same fifty, so its effect is measured, not assumed.
3. An LLM judge for what rules can't see. DeepEval scores each email against a written rubric, and a narrow custom check confirms each claim about the business against the record. The judge is Claude, because the drafts come from ChatGPT and a model shouldn't grade its own writing.

What the first eval run found

The suite ran over three weeks of history: 308 sent emails to 260 people, and the 208 leads behind them.

	Result
Emails passing every rule	90%
Leads passing every rule	78%
Leads emailed before the address was verified	22
A person I already talk to who got a cold email anyway	1
Businesses contacted through more than one person	16
Recipients emailed but never recorded in the CRM	5
Follow-ups	All 5–12 days apart; none after a reply
Placeholders, sensitive topics, "don't hire" claims	0
Bounces	2 from 308 sends

The writing was in good shape. The weaknesses were upstream, in verification, deduplication and tracking. That's exactly where reading every email could never have caught them.

Two lessons I didn't expect

My own check was the weakest link. My first website check compared the domain with the business's name on Google Maps. But the Maps listing comes from the same lookup as the website, so when the lookup goes wrong, both are wrong in the same way and agree with each other. A cleaning company matched to a cleaning marketplace "passed" because the Maps name and the website were both the marketplace. Comparing against the registered company name raised the catch rate from 2 of 10 wrong matches to 6 of 10.

A ready-made metric gave a made-up claim a perfect score. To test DeepEval's faithfulness metric, I added an invented sentence to a clean email: the business had "recently opened a second branch". It scored 1.0. By default the metric only penalises claims that contradict the source, and an invented fact contradicts nothing. Its strict mode swung too far the other way: it also failed my own sign-off lines, scoring the made-up email 0.17 and the clean one 0.14. I replaced it with a narrower check that looks only at claims about the business. It flagged the invented branch and passed the clean email, for under a cent per email.

Plant an error you know is there before you trust a metric. Mine didn't measure what its name suggested.

What it changes for the business

• Fewer wasted sends, earlier. Every unverified address, duplicate contact or wrong website the rules catch is a send that would have bounced, annoyed someone or been about the wrong company. They're now caught before the batch goes out, not discovered afterwards.
• My reputation stays intact. Cold-emailing someone I already talk to, or reaching the same business through three people, costs more than one lost lead. Those are now hard rules, not things I hope to remember.
• A healthy sending domain. Two bounces in 308 sends (0.6%) keeps the domain in good standing, and the verification rule keeps it there as volume grows.
• Changes become measurable. A new lane, prompt or enrichment step gets rerun against the same checks and golden set, so I see whether it helped instead of guessing.
• It's cheap. The rules are free. The judge costs about US$0.007 an email, around 35 cents a day at 50 emails. It took a day to build, and keeping it going takes a few minutes reading a daily report plus a 20-minute hand check of ten emails a week.

What the evals can't show yet is a change in replies. Three weeks of email replies is too small a sample to prove anything. The next measure is replies per 10 emails, by lane, before and after the upstream fixes.

Why I still send by hand

I could automate sending, and the 50-a-day goal makes it tempting. I don't, for the reason in the AI Trust Ladder: the system hasn't been right for long enough.

Evals are how that changes. "It's been right for long enough" used to be a feeling. Now it's a number: the share of emails that pass every check, week after week. When that number holds, sending is the next rung up the ladder.

What's next

1. Judge the full history. Grounding and quality scores for all 308 emails, as the baseline for the writing step.
2. ChatGPT vs Claude, head to head. Both draft for the same 20 businesses, a judge picks the better email, and I blind-score the same pairs to see whether the judge agrees with me.
3. Fix the upstream failures (verify before sending, one contact per business), rerun, and turn the 78% into a before-and-after number.