How Evals Cut Wrong Matches in AI Web Research by 83%

The system

At Flareworks, an internal production system (not a client deployment) researches Singapore small businesses and drafts around 50 cold emails a day. One of its lanes, Hiring Signals (the CRM agent system describes the rest), starts from public MyCareersFuture job ads: a firm paying a salary for admin and enquiry work is a firm with a visible operations problem. For each firm, the system finds its website, finds a verified email on that domain, and writes a short note grounded in what the firm does. Search and enrichment calls (Google Maps, Google Search, Hunter, email verification) run through treg, a pay-per-call tool catalog; small classification calls run on Jev, a compact judgment model from TypeSafe; drafting and evaluation use Claude.

The website is the anchor for every later step. The email address comes from its domain. The research and the personal line in the email come from its pages.

The problem: 1 in 5 websites belonged to someone else

The website step was simple: search Google Maps for the company name, take the first result, use its website. In the 2026-10-09 run it processed 570 firms and attached a website to 455.

A hand-labelled sample of 60 firms showed the cost of that shortcut:

• 12 of 50 websites belonged to a different business. A home-cleaning marketplace instead of a small cleaning company. A shopping mall instead of a tenant. A US freight company and a New York interior firm instead of Singapore ones. A car-sharing service that shared one word with an electrical contractor.
• 10 of those 12 firms had no website at all. The right answer was blank; the step guessed instead.
• 2 of 10 firms left blank did have a website the step never found.

Weighted to the whole run, roughly 1 in 5 leads with a website pointed at the wrong company.

Because the website is the first step, one wrong match compounds through every step after it:

1. The email address is found on the wrong domain, or on no real inbox at all. Bounces follow: an education centre's email went to a domain the rule check had already flagged as not matching the company name, and it bounced.
2. Bounces and spam reports lower the sender's reputation with mailbox providers, so later emails to the right firms are more likely to land in spam. Across 308 sends, the bounce rate was 0.6%; the system's value depends on keeping it there.
3. The personal line describes another business. A cold email's only advantage over a mass mailing is that it is about the reader. An email about the wrong company reads as spam, and it uses up the one contact the system will make with that firm.
4. Replies and follow-ups inherit the error. Any reply, follow-up or CRM record is built on the wrong research.

So a matching error is not a data-quality footnote: it sets the ceiling on deliverability, reply rate and the credibility of every later email. In this run the wrong matches were caught by manual review before sending, which put the cost on the reviewer: 20 flagged leads per campaign to check by hand.

The difficulty was that nothing looked broken. Every wrong website loaded, had a phone number and matched its Maps listing.

How the failure was understood

Why the evaluation started at the top of the pipeline

The pipeline runs in a fixed order: job ad → website → email → research → draft → send. No stage had reliable numbers yet, so the evaluation followed the same order, starting at the first stage and moving down.

The reason is proportional gain. Every later stage takes the website as given: an error there is copied into the email address, the research and the draft, and no later stage can correct it. Fixing the first stage raises the ceiling for all the stages below it. Measuring a downstream stage first would mean measuring it on inputs already known to be wrong 1 time in 5, so its numbers would mix its own errors with inherited ones. Clearing the errors from the start makes each later measurement clean.

Two links, only one of them checked

The step has two links: job-ad company → Maps listing → website. The second link is always true, because the website was copied from the listing. A wrong listing agrees perfectly with its own website, phone and address. Only the first link can fail, and nothing checked it.

The same error had already appeared in the evaluation suite: an early version of the website check compared the domain against the Maps name, which came from the same lookup. Recall on wrong matches went from 2 in 10 to 6 in 10 once the check compared against the registered company name instead. The rule that came out of both: a check must not share an input with the thing it checks.

Where the failures clustered

Of the 12 wrong websites, 9 came from job ads with no address. Without a postal code the search ran on the name alone, and the top result was whichever business ranked first for those words. The failure types:

Failure type	Example (anonymised)	Why the first result won
Marketplace or directory	Cleaning firm → cleaning marketplace	Marketplaces rank for every service name
Building, not tenant	Trading firm → the mall it sits in	Mall listing shares the postal code
Neighbour in the same building	Renovation firm → logistics firm next door	Same industrial-building postal code
Foreign lookalike	Logistics firm → US freight company	Similar name, no country check
Wrong URL in the job ad itself	Beauty distributor → a different brand's site	The employer typed it

How the benchmark was built

The golden set has three parts, because each answers a different question:

• 20 firms the system had flagged as possible mismatches. These measure how many wrong matches a check catches (recall). They cannot measure accuracy, because they were chosen for being suspicious.
• 30 random unflagged firms. These give the honest accuracy figure: 29 of 30 were right.
• 10 firms left with no website. These measure missed websites, which the first two parts cannot see.

Every row was labelled by hand against written rules, which took 1.5 hours for 60 firms. A website counts as the firm's own when one firm-tied signal matches: the UEN or registered name on the site, the same block, unit and postal code, the same phone or WhatsApp number, or the same founder. Group sites that list the firm count; directories, marketplaces and malls never do. A right website for a poor-fit lead is still a right website: fit is a separate question with its own test.

Precision, recall and a biased sample

The existing rule check (does the domain share a word with the registered name?) was scored on the same labels: it caught 7 of the 12 wrong websites (recall 58%), and 7 of its 11 flags were real (precision 64%). Its misses all shared one common word with the domain: cleaning, interior, blue, hock, butterfly. A common word is weak evidence, which is the same lesson as the matching step itself.

The system's own mismatch flag looked better, catching 11 of 12. That number is not usable: the 20 suspicious rows were selected by that flag, so it was always going to catch them. A detector cannot be scored on a sample chosen by the detector.

Two labelling rules came from cases where the first instinct was wrong. A salon trading under a different brand name was the right match (same address, unit and postal code). A specialist clinic inside a hospital building was a right match too, even though it is a weaker lead.

Evidence, not ranking

The model behind the fix: a search rank is not evidence. A candidate website counts only when something ties it to the registered company itself:

• the job ad gave the URL;
• the Maps listing carries the full registered name;
• the listing's postal code matches the job ad or the ACRA record and the names share a distinctive word or initials (postal codes alone fail in malls and industrial buildings);
• the site shows the UEN, the full registered name, or the postal code plus a name match.

Phone numbers from the listing don't count: they come from the same source as the candidate.

Calibrating the judge

Name-only matches go to Jev, TypeSafe's compact judgment model, for a yes/no score. It judged 11 such candidates per run: it rejected all 8 wrong ones and also both right ones, scoring them 0.70 and 0.40 against a 0.85 cut-off. Its scores moved by at most 0.04 between runs, so the judgment was stable; the cut-off was set too high. Lowering it would have fixed both on this set, but choosing a threshold from the test labels is tuning to the test, so it was left unchanged. A judge needs two checks: consistency across runs, and agreement with human labels at the threshold actually used.

The rebuilt matching step

The matching step was rebuilt as one function, tested on its own:

1. Registered address by UEN from the free ACRA dataset on data.gov.sg (includes sole proprietors).
2. Candidates: the job ad's URL plus the top three Maps results, searched with the job-ad or ACRA postal code.
3. Score each candidate on the firm-tied signals above. Directories, job boards and marketplaces are set aside as research links, never used as the website or for email.
4. Decide: one firm-tied site → use it. Several → prefer the one showing the UEN or registered name, otherwise flag for a person. Name-only matches go to Jev for a yes/no judgment. Nothing → one Google search under the same rules → otherwise leave blank.

Blank is a valid answer. A missing website costs one lead; a wrong one costs an email to the wrong business.

Searches and web pages are cached, so every test run sees identical inputs and only the judgment can vary. Each variant ran three times.

Results

	Before	After: benchmark (60)	After: fresh holdout (23)
Wrong website	12/60 (20%)	2/60 (3%)	0/23 (95% bound: under 13%)
Right outcome, no human input	46/60 (77%)	55/60 (92%)	19/23 (83%)
Right outcome after a person picks among flagged options			22/23 (96%)

Wrong-company websites went from 1 in 5 to none in 23 fresh leads: an 83% cut on the benchmark (12 to 2), and 64% fewer errors of any kind (14 to 5). The two "after" columns are not directly comparable: the benchmark was deliberately loaded with suspicious firms, while the holdout is the top-scored leads production would email. The wrong-website count is the fair comparison across both. The whole evaluation cost 1.5 hours of labelling and about US$0.30 in search and enrichment calls.

On the same 60 labelled firms:

	Before	After
Wrong website given	12	2
Right websites kept (of 38)	38	36
"No website" firms correctly left blank (of 10)	0	10
Missed websites found (of 2)	0	1
Right outcome (of 60)	46	55

Search cost for the full benchmark was about US$0.05.

Does a bigger model help?

The same frozen candidates were given to four Claude models, each asked blind which candidate (if any) is the firm's own site. Three runs each:

Judge	Wrong	Missed	Right (of 60)	Same answer in all 3 runs
Rules + Jev (TypeSafe)	2	3	55	60/60
Claude Haiku	1–2	5–6	53	57/60
Claude Sonnet	2–3	1–2	56	59/60
Claude Opus	2	2	56	60/60
Claude Fable	2	3	55	60/60

Each model ran as a separate blind session on the same frozen dossier (60 firms, 103 candidate sites), so differences come from judgment alone. The three larger models and the rules land within one firm of each other. The jump from 46 to 55 came from what the judge was shown (firm-tied evidence, a clean candidate list), not from the size of the model. The smallest model was too cautious, leaving real websites blank.

Overfitting, and a fresh holdout

The rules were refined over four rounds on these 60 firms. Each round fixed a class of failure rather than a firm (postal codes need a name match; mall listings are not evidence; names made only of common words match on the whole name), but any set used for tuning flatters the result. 55 of 60 is an optimistic figure.

The holdout: a new weekly run on 2026-10-21 produced 244 leads, 208 of them new to both the tuning set and the CRM. The top 25 by lead score, the ones production would actually email, went through the matching step once, live, with no changes afterwards, and were labelled by hand:

Outcome	Firms
Right website	17
Correctly left blank	2
Flagged for a person, right site among the options	3
Missed a real site	1
Wrong website	0
Label unsure (not scored)	2

19 of 23 were right with no human input, 22 of 23 after a person picked among flagged options, and none were wrong. Zero in 23 is not proof of zero: by the rule of three, the true wrong-site rate is below about 13% at 95% confidence. Each further batch narrows that bound. The holdout also surfaced a case the tuning set did not have: two sister companies, sharing a registered office, received the same website. The matching was right; the duplicate belongs to the dedupe step.

Opinion: the system has to be able to say "I don't know"

This paragraph is a view, not a finding. The largest single gain in this work came from allowing the system to return nothing. Ten of the twelve original errors were firms with no website at all, and the step filled the gap with whatever ranked first. A research agent that must always produce an answer will produce a confident wrong one exactly when the evidence is missing, and those are the cases that cost the most downstream. "No website found", "several possible websites, please choose" and "not sure" are outputs to design for, measure and reward, not failures to hide.

What the benchmark showed

• A check that shares an input with the step it checks will agree with that step's mistakes.
• A suspicious sample measures recall; only a random sample measures accuracy; only a sample of blanks measures misses.
• Postal codes identify buildings, not businesses.
• Leaving a field blank is better than filling it wrong.
• A bigger model did not beat better evidence: Sonnet, Opus, Fable and the rules scored within one firm of each other.
• Only a holdout the rules never saw gives a number worth publishing.