Our AI matched our answer key at 0.75. Then two of us filled in the same answer key, separately — and matched each other on about half. The model agrees with us better than we agree with ourselves.
This is the story of how that happened and what we did about it. It ends with the math we now use to check whether a measuring stick can be trusted — a formula ecologists use to count species in a forest, written by a Bletchley Park codebreaker. If you've ever labeled your own work "for AI" and ended up with mush, the mush probably wasn't the model's fault. It wasn't in our case.
The job: marking work
We map business processes to see which steps an AI agent can take. The method is simple to describe. Take a process — contract review, month-end close, handling a customer complaint. Split it into steps. For each step, mark the capabilities it needs: extracting data from a document, checking terms against a playbook, judging which risk is acceptable, negotiating with another party. Our public list has seventeen capabilities — thirteen cognitive, four physical ones that mark the boundary of what software can touch (the list is at hrpulsar.com/methodology/capabilities#list).
The marks add up to a verdict. Some capabilities are gates: a risk call, an open-ended conversation, weighing people against each other for an assignment. If a step carries even one gate, it stays with a human, no matter how well an agent handles the rest. Checking against a playbook — an agent does that today. Deciding which clause to concede in a negotiation — a gate.
Companies read the verdicts to decide where agents go first, so a wrong mark is not a cosmetic bug. A false gate hides automatable work. A missed gate is worse: it shows "an agent can take this" on a step where someone has to answer for the outcome.
To trust the verdicts, we keep an answer key: a library of 39 processes, 390 steps, marked by hand, that every model run is scored against. The question this article answers: who checks the answer key?
The model wasn't the problem
The symptom that started the investigation was instability. Run the same process through the model twice and every third step came back with a different set of marks — 64 to 67 rows out of 104 matched between runs, the rest didn't. Our first instinct was the usual one: the model is sloppy, get a better model.
We tried a stronger model. The results got worse, and they got worse in a way that pointed somewhere: the stronger model read our rules more literally, and where the rules contradicted each other, it faithfully reproduced the contradiction. That was the first hint the problem was on our side of the keyboard.
So before blaming any model, we tested ourselves. Two of us marked the same processes independently, reading the same rules. Then we put the sheets side by side and counted. About half the rows matched. The precise number was a micro-F1 of 0.47 — one line of arithmetic: agreement = matched marks, counted twice, divided by all marks both people made. Nobody was careless; both sheets were defensible readings of the written rules.
The diagnosis, once we looked, was embarrassingly ordinary. Our marking rules lived in five documents: a glossary of capabilities, the rules for breaking down a process, the rules for assessing people's skills, the worked examples we feed the model, and the answer key itself. Written at different times, for different purposes, by people who each remembered the conventions a little differently. The glossary called escalation one thing; the process rules called it another. The examples marked a step one way; the answer key marked its twin the other way. Each of us — and each model run — resolved the contradictions in its own direction.
We merged everything into one rulebook: one definition per capability, a "mark it when" line, a "don't mark it when" line, examples that agree with both. Then we re-marked our own answer key under the merged rules. 81 rows out of 390 changed. A fifth of our gold standard was not gold; it was five documents arguing.
After the merge, the model's accuracy against the answer key did not move: 0.76 before, 0.75 after — inside the noise. If you were hoping a cleaner rulebook makes the model smarter, it doesn't. The model was never the bottleneck. What did move was the quality of the answers people see. False gates — steps wrongly marked "needs a human" — dropped by half, for both models we measure (11 to 5 extra risk calls per run on one, 16 to 9 on the other). The headline verdict, "who closes this step," now disagrees with the answer key on 14–16 steps per hundred instead of 20.
One example beats a paragraph of rules
The sharpest lesson came from a single case. Steps where the conversation itself decides what to concede — negotiating a refund, agreeing on a partial delivery — need a human by our own rules. The model missed that gate on every such step, in every run: zero out of nine. The written rule was fine. The worked example we fed the model told a different story: it showed a negotiation marked without the gate, because in that example the concession had been decided a step earlier. The model, reasonably, generalized the example over the rule.
We fixed the rule's wording and the example. The model now marks the human decision in eight runs out of nine. From 0/9 to 8/9, and the change that mattered was not the paragraph of rules — it was the example. If your prompt has both, test them separately: your examples are rules, and they outvote the text.
The expensive side of the error
One honest number against ourselves. While false gates halved, dangerous misses went the other way: 4–7 per run after the merge, against 1–3 before. A dangerous miss is a step shown as agent-ready when a human should answer for it — triaging patients in an emergency department, declaring the severity level of an incident. Tightening the rules against over-marking pushed some errors to the expensive side.
This is the asymmetry anyone building verdicts on labels has to respect: the verdict follows the worst mark. One missed gate flips a step from "human" to "agent" — and the cost of that flip is not symmetric with the cost of a false gate. A false gate wastes an opportunity; a missed gate hands someone's responsibility to a system that can't carry it. Decision theory has a plain instruction here: when the two error costs differ, don't put your threshold in the middle. We now treat "agent can take it" as the claim that needs the stronger evidence. (How to check that the humans reviewing those steps are actually reviewing — that arithmetic is in an earlier article: https://hrpulsar.com/blog/1472987137-how-to-check-the-reviewer-is-still-reviewing.)
How many capabilities does work have?
The practical half of this story ends there: one rulebook, measurable agreement, errors sorted by cost. The scientific half starts with a question we couldn't answer with engineering alone: is the list of seventeen capabilities complete? If real work keeps producing capability number eighteen, our answer key is a moving target and every verdict has an expiry date.
Ecology has been asking this exact question for a century — how many species live in the forest, given that you've only caught some of them — and it has tools we could borrow whole.
The first tool is the accumulation curve: walk through the catalog in the order we built it and count when each capability first appeared. Our first process produced 7 capabilities. By process five, 10. By process twenty-seven, all 17. Then the curve goes flat: the last 12 processes — 135 steps across engineering, sales, support, media — added nothing new.
The second tool is the Chao1 estimate, one line: true richness ≈ observed + (singletons² ÷ 2 × doubletons), where a singleton is a capability seen exactly once. The logic in words: if your sample is full of things you've seen only once, many more are hiding; if even your rarest find shows up repeatedly, the inventory is probably complete. Our rarest capability appears 8 times across 590 marks; the rarest by process shows up in three different processes. Singletons: zero. Chao1 says the list is as long as what we've already seen — no hidden tail.
The third tool is the one from Bletchley Park. I. J. Good, Alan Turing's statistical assistant, published the estimator they used against Enigma: the probability that your next observation is a species you've never seen ≈ the share of observations that were singletons. Ours is 0 ÷ 590 — zero. A suspiciously clean zero, so let's be honest about it: a zero estimate here means "nothing in the data argues for more," not "more is impossible." The honest upper bound is the rule of three: after 135 steps with no new capability, the 95% confidence bound on the next step producing one is 3 ÷ 135 ≈ 2.2%. Per process: at most 25%. And one caveat belongs in print: a capability enters our catalog only after showing up in three processes, which partly explains the absence of singletons — though the queue of pending candidates is also empty.
So: seventeen capabilities, a flat curve, and a bounded risk that the next process breaks the list. For a reader doing this at home, the recipe is three lines — order your sample, plot first appearances, count your singletons. If you have many, your catalog isn't done; if you have none, stop adding categories and start measuring.
The sample is what it is — 39 processes and 390 steps from one library is a lot of marking and a small slice of the economy — so these are our numbers, not laws of work.
Where the boundary runs
One more thing the catalog let us test. Conversations split into two capabilities: dialogue by script (an agent handles it) and open-ended dialogue (a gate). The criterion we settled on is a single question: is the set of outcomes known before the conversation starts? A support ticket walks a tree of known resolutions — script. Negotiating what to concede when stock runs short — open.
If the criterion is real, the two marks should never need to sit on the same step. Across all 390 steps: 37 steps carry a conversation mark, zero carry both. The same holds for the neighbouring pair — coordinating people by lists and schedules versus weighing people's input against each other: 46 steps, zero overlap. A boundary you can state as one question, and that survives 390 real steps without an exception, is a category you can build on. Task frameworks in economics have needed categories like this since Autor, Levy and Murnane's 2003 paper. One difference matters to us: our unit is a step plus who answers for it, not a task in isolation. A step can carry no capability at all and still be unautomatable, because signing is responsibility, not skill.
The ceiling
Back to the two numbers in the first line. If two trained humans agree at 0.5, what does "the model scores 0.75 against the answer key" even mean? Two explanations, both partly true. First: the model learned the answer key's dialect through our worked examples — it agrees with the document, not with some deeper truth. Second: the answer key itself is one person's resolution of ambiguities that another person resolves differently. Either way, the conclusion is the same: human agreement is the ceiling of measurable accuracy. Below that ceiling, you're not measuring the model; you're measuring whose contradictions it absorbed. In our data, rows where every source disagrees with the answer key together account for up to 0.08 of the model's apparent error — labeling debt, not model error.
The research community is starting to hit the same wall. The best-known study of AI exposure, Eloundou and colleagues' "GPTs are GPTs" (2023), labeled 19,265 work tasks with both humans and GPT-4 and reported about 81% agreement between them. The number I couldn't find in the paper is human–human agreement on the same labels. That's not a criticism — it's the measurement everyone skips, including us until last month. It belongs in every labeling study's first table. (We also ran a very different comparison — a large reasoning model against small fixed-form decision models on this same marking task — and neither passed our bar; that story is here: https://hrpulsar.com/blog/1469513729-the-question-picks-the-model-jev-and-claude-on-three-of-our-tasks.)
The pattern
If you're marking your own processes "for AI," steal the procedure, not our numbers:
Before you blame the model, measure yourselves. Two people, the same thirty steps, independent marks, one afternoon. Count the agreement. That number is the ceiling of any model accuracy you will ever report.
Run the model twice. The rows that differ between runs are a map of where your rules are silent. Fix the rules, not the temperature.
Test your examples separately from your rules. Examples outvote text.
Count your singletons before trusting your categories. Zero singletons and a flat curve — your list is done. Many singletons — keep collecting before you measure anything against it.
Measurement starts with people agreeing. Five hundred years ago, surveyors knew it: Köbel's 1535 textbook tells you to build the lawful rod by lining up sixteen men's feet, heel to toe, as they leave church — sixteen different feet, averaged into one lawful unit. Our answer key is the same sixteen feet. So is yours.
Agree on the ruler. Then measure.
If two people on your team ever marked the same work independently — what agreement did you get? I'll trade numbers: ours was 0.47.