The Answer Got Cheap. Checking It Didn’t.

Generation stopped being the hard part. In data analysis, that changes which questions are worth asking — and which tools are worth buying.

Ask any system for an answer today and one arrives. It is fluent, formatted, and confident. Then comes the pause that nobody had budgeted for — the moment where a person has to decide whether to believe it, and discovers that the tool which produced the answer in four seconds has nothing to say about whether the answer is right.

That pause is where the AI era is currently stuck. Not on generation, which is solved to a degree that would have seemed absurd three years ago, but on the far less glamorous question of what to do with abundant output nobody has time to check.

The economists have started naming it. In Some Simple Economics of AGI, Christian Catalini, Xiang Hui and Jane Wu model the transition as “the collision of two racing cost curves: an exponentially decaying Cost to Automate and a biologically bottlenecked Cost to Verify.” Their conclusion is blunt: “The binding constraint on growth is no longer intelligence but human verification bandwidth.” Producing scales with compute. Checking scales with people, and people do not scale.

The inversion

For as long as software has existed, the expensive half of analytical work was production. Getting the data, joining it, shaping it into something that answered the question. Verification was a rounding error by comparison — when a person spent three days building an analysis, they understood it in their bones by the time it was finished. Comprehension came bundled with construction, free of charge.

That bundle has come apart, and the collapse inverts the economics of the whole workflow. The construction is now instant, and the comprehension that used to arrive with it does not. What was once a byproduct of doing the work is now a separate task, performed on unfamiliar output, by someone who did not build it.

The bottleneck moved. Most tooling has not.

Why code noticed first

Software development has been living in this future for two years, and it has the scar tissue to show for it. The most rigorous measurement to date comes from METR, which ran a randomized controlled trial on experienced open-source developers working in repositories they knew well. The finding, verbatim from the paper: “Surprisingly, we find that allowing AI actually increases completion time by 19% — AI tooling slowed developers down.”

The more revealing number is the one that came after. Going in, the developers expected AI to speed them up by 24%. Having finished the tasks and been measurably slowed, they still believed it had sped them up by 20%. The experience of working faster survived contact with evidence that they hadn’t.

Restraint is warranted in reading that: it is sixteen developers across 246 tasks, on mature codebases they already understood, in early 2025. It is not a universal law, and the authors say so. But it is the clearest available picture of the shape of the problem — the gap between how fast the work felt and how fast it was, which is exactly the gap verification opens up.

Every one of those findings is about code. The honest version goes one step further than any of them do.

Analysis fails quietly

Code has a property that analysis lacks: it fails loudly. A wrong function throws an exception. A bad type breaks the build. A regression trips a test, and if it survives all of that, it eventually produces a page at three in the morning. The environment is adversarial by construction, and the errors announce themselves.

A wrong number announces nothing. It renders identically to a right one — same font, same decimal places, same air of authority. It flows into the deck, gets presented, and becomes the basis of a decision. Nothing crashes. The reporting line for a wrong analytical conclusion is a business outcome six months later, by which point nobody is auditing the number that started it.

Which means the verification problem is not merely present in data analysis. It is present without the feedback that made it visible in code. Software got a warning. Analytics gets silence.

There is a reason this stayed dormant for so long. Analytics never had a generation boom, because the queue was the rate limiter — questions were rationed by the availability of someone who could answer them. And that person, by doing the work, arrived at the answer holding the reasoning that produced it. The check came attached. Remove the queue, and the check does not automatically come with what replaces it.

What checking actually costs

The instinctive fix is provenance: show the sources, cite the rows, display the query. This is worth doing, and it is not verification.

Provenance establishes that the arithmetic ran on real data. It says nothing about whether the arithmetic answered the question. A number can be perfectly traceable and still be the answer to something nobody asked — the right calculation over the wrong window, a segment silently dropped, a definition of “active user” that made sense to the system and would not have survived ten seconds of contact with the person who asked.

Verification means following the reasoning: what the question was decomposed into, what was filtered and why, what was counted and what was deliberately not. The choices before the query, which is where analytical work is actually decided.

That is also what makes the cost asymmetry possible. Rebuilding those choices from scratch is what makes redoing an analysis expensive. Reading them, when they are stated explicitly, is a different and cheaper act — the difference between finding a proof and reading one. Reading a proof is cheaper. It is not free, and any tool claiming otherwise is selling something.

Consider a team told that conversion fell among users arriving from a particular channel. Verification is not confirming the percentage. It is asking whether that cohort was defined the way this team defines it, whether the window covers a period when something else changed, whether the comparison group is comparable at all. Those questions are answerable in a minute if the system shows its reasoning, and require rebuilding the entire analysis if it does not. Same answer, same number — an order of magnitude apart in what it costs to trust.

The stronger objection is that verification will simply be automated too. Semantic layers already encode definitions; lineage and tests already catch some silent failures; a system can run its own consistency checks. Much of that is real, and it genuinely shrinks the burden — the mechanical part of checking is as automatable as the mechanical part of producing. But it moves the boundary rather than removing it. Every one of those mechanisms verifies that the analysis was executed correctly. None establishes that it was the right question, defined the right way, for a decision the system was never told about. That residue is small, and it is exactly the part a decision turns on.

The limit worth stating

There is an uncomfortable corollary, and pieces that skip it are not being honest about the problem.

Cheap verification requires judgment in the reader. Someone who does not know that a particular attribution channel is unreliable will not flag it, however legibly the reasoning is displayed. Legibility lowers the cost of checking for a person equipped to check. It does not manufacture the equipment.

So the benefit lands unevenly, and it lands most on the person who needed the least help. That is the real shape of the thing, and it points somewhere more interesting than where this conversation usually goes.

If verification is the binding constraint, then the scarce resource is not the ability to produce analysis. It is the judgment to interrogate it — and that judgment belongs to the people who have spent years developing it. Their work moves up the stack: less time assembling the answer, more time on whether it holds. The analyst stops being the bottleneck on production and becomes the reason the output is worth anything. That is not a smaller job. It is the part of the job that was always the point, finally unburdened by the plumbing that used to consume the week.

The capability spreads outward at the same time. People who never had an analyst get a floor they never had. But a floor is not a ceiling, and the ceiling is still made of judgment.

The question that sorts the market

The next generation of analytics tools will not be sorted by whether they can produce an answer. That has become the easy part, and easy parts do not differentiate.

They will be sorted by what happens in the pause afterwards — whether the person on the other end can establish that the answer holds for meaningfully less than it would cost to produce it again. A tool that halves the time to an answer and leaves verification untouched has not removed the bottleneck. It has moved it downstream, onto a person, where it is harder to see and easier to ignore.

The evidence layer stops being a feature and becomes the product’s load-bearing wall. Charts and tables do not disappear in this world; their role changes. The chart is the proof. The conclusion is the product.

Catalini and his co-authors call the widening space between what systems can execute and what humans can afford to check a “Measurability Gap.” In analysis, that gap has a specific and unglamorous form: the distance between an answer arriving and anyone being sure of it. Closing that distance is the work.

Key Takeaways

• The cost of producing an answer collapsed. The cost of checking one is bounded by human attention, and did not move.
• Code fails loudly and analysis fails quietly — which makes the verification gap harder to see in data, not smaller.
• Provenance is not verification. Knowing where a number came from is not knowing whether it answered your question.
• The test for any AI analytics tool: can the answer be checked for meaningfully less than it cost to produce?
• When verification is the constraint, judgment becomes the scarce asset — and the people who have it become more load-bearing, not less.

The economic framing is drawn from Christian Catalini, Xiang Hui and Jane Wu, “Some Simple Economics of AGI” (arXiv preprint, February 2026). The developer productivity figures are from METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” (arXiv, July 2025).