An AI quality gate for every article, and the six it should have failed

What the research says about AI judges

The 2023 G-Eval paper (Liu et al., Microsoft) tested a simple idea: ask a strong language model to grade text against written criteria, and weight its score by how confident the model is in each possible answer. On a standard summarisation benchmark, that method agreed with human ratings more closely than any earlier automatic metric, with a Spearman correlation of 0.514. The same paper raised a warning: AI judges may favour text written by AI.

DeepEval, an open-source evaluation library from Confident AI, turns that research into engineering practice. Three of its recommendations matter here:

1. Fix the scale. Give the judge explicit score bands, so it can't drift towards the middle.
2. Break a judgment into small decisions. Its DAG metric runs pass-or-fail gates in a fixed order, with scores assigned in advance, and only then grades quality.
3. Check the judge against people. Keep a set of examples with known answers ("goldens"), and track whether the judge still agrees with them after every change.

This site's publishing gate already did the first. This article tests it against the third.

How the gate works

Every article lives as a file in the site's code repository. Before publishing, a script sends each one to Jev, a small AI model that answers narrow questions with a probability for each answer instead of free text. It asks twelve questions:

• Substance: is the main claim specific and arguable, is the evidence first-hand, will the reader recognise their own problem?
• Delivery: does the opening create tension, does every paragraph earn its place, is the ending the sharpest part?
• Fit for the reader: is it plain enough for them, and does it leave a practical takeaway?
• Penalties: does it overclaim, write for the wrong reader, borrow an idea without credit, or fail to deliver what the title promises?

Plain code, not the model, turns those answers into a score out of 10. Weak substance or weak delivery caps the score just below the bar, whatever the other answers say. Anything with an exact answer, such as title length, missing image text or a broken link, is checked by plain code as well.

Each score is saved with a fingerprint of the article's text. A commit hook blocks any published article that scores below 8, or that has been edited since it was scored, so an article can't be published on an old score. 32 articles are published, scoring between 7.1 and 9.6. Company profiles have a lower bar of 7.

Test 1: does the score hold still?

Six published articles were each scored five times. The largest gap between one article's runs was 0.2 points; three articles scored within 0.1 point every time, and two scored identically all five times. The judge is consistent. Asking twice and taking the median would add cost without changing a decision.

Consistency does not show the judge is right. That needed the second test.

Test 2: does it agree with the author?

The site's git history holds a record of the author's own judgments: versions of articles that were rejected and rewritten. Seven of them made a usable test set, each scored against the reader the article is written for.

Rejected version	Why it was rejected	Score	Gate
The Modern Shrine (original)	Weak model section, generic claims	5.3	Failed
MummyReady	A section addressed to Forward Deployed Engineer recruiters, ending "DM me"	9.1	Passed
AI-native operations	A CV-style skills list and a "compare notes" sign-off	9.1	Passed
Mindreader product pivot	A CV-style skills list	9.0	Passed
LangChain content pipeline	"The relevant skill was…" written to a hiring reader	9.0	Passed
Forward-deployed AI	A CV-style skills list	8.6	Passed
Private search engine	A CV-style skills list	8.2	Passed

The gate failed the article that was badly written. It passed the six that were well written but spoke to the wrong person. Their approved rewrites scored about the same, between 8.3 and 9.7.

That is not a broken judge. It is a judge doing a different job. The questions ask whether the writing is strong. None of them asks whether a section sounds like an internal note, and a skills list written for a recruiter can be perfectly clear, specific and tight.

Test 3: one narrow question

Following DeepEval's advice to split judgments into small decisions, one new yes-or-no question was put to the same seven pairs: does any section read as an internal or job-application note rather than a public article, for example by addressing a recruiter directly, listing the author's skills like a CV, or asking the reader to DM?

• MummyReady: 0.85 before the rewrite, 0.26 after.
• AI-native operations: 0.81 before, 0.08 after.
• The four skills-list pages: between 0.12 and 0.38, before and after alike.
• Every approved version: 0.38 or lower.

A cut-off at 0.6 would have caught the two most obvious cases with no false alarms on approved writing. It missed the quieter ones, where the only problem was a list of skills. Those were caught by a person reading the site.

What changes in the gate

The tests point to three changes. The first has shipped; the other two are next.

1. A loophole was closed. Changing an article's intended reader didn't invalidate its old score, because the fingerprint covered only the words. It now records the reader and the article type too, and the hook flags any mismatch.
2. Structural checks before the score. The narrow question becomes a gate of its own, in the order DeepEval's DAG metric uses: fail on structure first, then grade quality.
3. The seven rejected versions become a permanent test set. Any change to the questions has to keep failing them where the gate should, so a rubric change is judged on evidence rather than opinion.

One gap remains. The gate does not check facts. Corrections such as 900 learners rather than 1,000 came from checking each number against its source by hand. Checking each claim against a record of sources, the way DeepEval's faithfulness metric checks an answer against its documents, is a larger piece of work. The same separation of writing and grading runs in a chatbot that grades every answer before a customer sees it.