5,023 Articles, 9 Published

Twenty-three defects between a filter and an editor

Last week a filter of ours passed 672 articles as credible. It ranked none of them.

That sentence is the whole problem with most AI projects, and it took a measurement to see it.

Here is the funnel, honestly:

The first three rows were built in a day. They are scoring, thresholds and a source-quality table — ordinary work, and AI coding tools make it faster every month. Implementation stopped being the bottleneck.

The last two rows are the product. And they are where the twenty-three defects were.

The filter had nothing left to say

Of the 672 credible candidates, 390 — 58% — scored a perfect 21 out of 21 on their best decision axis. Another 6% scored 20.

A sort on a saturated key is not a ranking. It is the insertion order of the database wearing a suit. Raising the bar to 21 would still leave 390 items tied. There was no threshold to tune, because the scale had stopped discriminating at exactly the point where discrimination was the entire job.

The second gate failed differently. 53% of the "credible Tier A/B" pool came from open publishing platforms — papers.cool 97, arXiv 74, Medium 64, YouTube 36, HackerNoon 23. The quality table rates the feed, not the author. "Artificial Intelligence on Medium" is Tier A and carries thousands of unrelated writers. So an anonymous Medium post sat next to the WSJ at equal credibility, both at 21/21.

Two gates, both passing, neither deciding anything.

What the human baseline cost, and what it bought

A person read 96 articles and kept 9. That is 1.3% of the credible pool.

The part worth recording: three of those nine verdicts changed after reading the article body rather than its summary.

• A Medium post on medical AI read as opinion. The body contained an NYU Langone study — 12 physicians, 1,800 blind-scored answers, full numbers. Rejected on the summary, kept on the content.
• A piece on contract review turned decisive on one sentence: the study's answer key had been built using the answers of the AI being evaluated. Absent from the summary.
• A supply-chain piece rested on a single contrast — five days for the agent, eighteen months to reconcile two dates — that only the body gave.

The cost of reading the body instead of the summary is a batch of ten going from ~4K to ~20K input tokens. That was the single highest-leverage decision in the system, and it was invisible until someone measured what the summaries were throwing away.

Then the system matched 8 of the 9

We rebuilt the same judgment as a pipeline: candidates, judge, edition, write, render. On a comparable week it ran 803 candidates → 485 credible → 143 admitted → 8 published, in 1 minute 52, in French and English.

It reproduced 8 of the 9 human picks.

It cut the ninth on the axis that asks whether the item changes what the reader does. That cut was correct. The item had been kept to avoid leaving a section empty — a quota, not a judgment. The system was stricter than the person who trained it.

That is the useful result, and it is not "the model is smart." The model was the same model on day one, when the first run never even showed the judge 6 of the 9 correct answers.

Where the twenty-three defects actually lived

Four patterns, and they generalize past our case.

Sampling that only existed because a human gets tired. The first run judged a sample, because 96 articles is what a person can read. The machine judges all 481 in ninety seconds. We had encoded a human limit as a system constraint, then measured the system against the human it was imitating. Removing the sample recovered most of the missing picks.

Every scale saturates. The angle score saturated at 21/21. Then the new judge's own scale saturated — 66% of admitted items scored 14 or 15 out of 15, and one axis awarded full marks to 51% of the corpus. Re-anchoring the rubric on scarcity ("3 means among the best this week, not 'it qualifies'") took admissions from 143 to roughly 95 and made the order an order again. A second axis, which was treating "quotes a regulator" as equal to "measured this," was capped: admissions 376 → 254.

Asking the model was not enough. Five quality mechanisms had to be moved out of the prompt and into code, each after watching the prompt version fail:

• Story clustering — single linkage produced one 46-item blob swallowing Waymo, Databricks and a water-treatment cyberattack. Complete linkage at 0.82 was the only threshold that also kept two separate OpenAI/Anthropic incidents apart, at 0.806.
• Approval voting — one editor picking 9 from 70 was pure variance. Three independent ballots, ranked by votes then mean rank.
• Composition caps — the admitted pool is dominated by security, so a faithful vote produced six of nine items on one theme and four empty sections. Caps of two per section and two per theme, enforced in code.
• Voice control — a checker for superlatives, hedges, clickbait and jargon. One violation triggers a rewrite naming the fault; if it persists the item is replaced, never patched.
• Quote verification — the shareable line now quotes the source verbatim, and the quote is checked against the article body. The model had marked three as verbatim that were not, translated two, and let one restate the headline.

And the model will invent the one thing you did not give it. A subject line wrote "take effect in 24 months" when the item said "from August 2026" — because the subject-line prompt received the judge's score and not the content. It now writes from the finished copy, forbidden to add a date, a figure or a deadline.

What is still wrong

Two runs share 6 of 9 items, up from 5. The chain is stochastic end to end; the judge admits between 91 and 100 items depending on the run. What varies is the tail, not the head — the AI Act item, the non-transferable contract and Mastercard come out every time — but it varies.

Three sections came out empty this week. If one of them is empty every week, that is a tagging defect, not rigor.

And nothing is deployed. No sending, no qualifying signup, no public archive.

The part that transfers

Nobody in this project was short of engineering. The filter was quick and it was correct. It also decided nothing, and it would have shipped looking like it worked — 672 credible candidates, sorted, a number on every row.

What made the difference was writing down what a good call looks like, measuring the system against a baseline someone actually produced by hand, and then moving each rule that mattered out of the prompt and into code where it could not be talked out of.

That knowledge existed before any of this. It sat in one person's head and in a specification nobody had formalized. No model had seen it, and no vendor could have reached it.

If AI should be paying off somewhere in your operation and the numbers do not show it yet, the gap is usually the same shape: the work runs on judgment that has never been written down, and the pilot measured itself against nothing.

Bring us the work that should be paying off.