Usage Is the Vanity Metric of the Intelligence Era

A tool earns product-market fit when people come back. Something sold as intelligence has to clear a harder bar — and most teams are still measuring the easy one.

There’s a number every software team learns to watch: retention. Did they come back? Came back last week, came back this week, came back enough weeks in a row that you can draw a flattening curve and call it fit. For a tool, that curve is the truth. A tool’s whole job is to be there when you return to it, so a return is the product working as designed.

That instinct is now quietly misleading a lot of teams building with AI. Because if what you’re selling isn’t a tool but intelligence — an answer, a judgment, a thing that’s supposed to change what someone does — then the return visit stops being proof. A person can open your product every morning, read what it says, and decide exactly what they would have decided anyway. High retention. Zero fit. They came back out of habit, or hope, or because the dashboard is bookmarked. None of that means the intelligence landed.

This is the part the borrowed metric hides. Usage is the vanity metric of the intelligence era — easy to grow, easy to celebrate, and silent on the only question that matters: did the intelligence change a decision?

Why the old metric stops working

For most of software history, the metric and the value were close to aligned. The tool’s value was the using of it — the spreadsheet open, the dashboard refreshed, the report pulled. So a return visit was a reasonable stand-in for value delivered: people came back because they were getting something, and “came back” was easy to count while “got something” was not. The proxy held because returning and benefiting mostly traveled together.

Intelligence breaks that alignment. The value of an answer isn’t in being read; it’s in what it changes downstream. You can consume a brilliant analysis and do nothing with it, and you can get one sharp answer once and reorganize a quarter around it. The act of using and the act of benefiting have come apart — which means the metric that conflated them no longer measures fit. It measures the habit of showing up.

The market has started to notice the first half of this. Andreessen Horowitz, looking at AI retention data, warns that “with the surge of AI tourists who sign up and churn after a couple months, early curves aren’t as indicative of long-term retention” — traditional SaaS benchmarks, in their words, may not apply directly. Bessemer puts the same caution more bluntly for founders chasing usage spikes: “Novelty isn’t the same as value.” Both are saying that healthy-looking usage can be experimentation rather than dependence, and that the old benchmarks mislead.

That’s the first half. The honest version goes one step further than either: for an intelligence product, even real, durable usage — the kind that survives the tourist churn and settles into a habit — still isn’t the finish line. It’s necessary and not sufficient. The finish line is a changed decision.

The bar we actually hold

So if usage isn’t the signal, what is? Here are the three we watch — none of them a login count.

The last answer generates the next question. Curiosity becomes self-generating. Nobody has to send a re-engagement email, because the answer the system just gave raised a question the person now has to ask. When a product stops needing to remind people to come back — when the work itself pulls them forward — that’s not retention you engineered, it’s retention the intelligence created.

The intelligence changes a decision upstream of the product. Someone ships a different feature, kills a roadmap item, reallocates a channel, reorders a priority — and can name the answer that moved them. This is the hard one to measure and the only one that counts. It means the intelligence didn’t just inform the work; it altered it.

The test for this signal is uncomfortable and clarifying: ask whether the team would have made the same call without the answer. If a system tells you activation is down and you already knew that and already planned to look at onboarding, it informed you but it didn’t move you. If instead it surfaces that the drop is concentrated in the one segment you’d just decided to invest in — and the roadmap changes because of it — that’s a decision that wouldn’t have happened otherwise. The difference between those two is the difference between a product people consult and a product people decide with, and only the second has fit. Most analytics, however heavily used, never crosses that line; it confirms what the room already suspected and lets everyone feel informed.

Going back becomes intolerable, not just inconvenient. Not “this is handy” but “I can’t go back to dashboards.” The baseline expectation has reset. Once a team has had the answer delivered, waiting a week for an analyst to build a view feels not slow but broken. Irreversibility is the strongest signal there is, because it’s the one users can’t fake and you can’t manufacture.

Notice what these have in common: not one of them is satisfied by someone simply opening the product. Each requires the intelligence to have done something — provoked, changed, reset. That’s the bar. It’s harder than retention on purpose, because it’s the bar that separates a thing people use from a thing people decide with.

The objection, met

The reasonable pushback is that retention is the proven PMF signal, validated across thousands of companies, and that throwing it out for something as soft as “did it change a decision” trades rigor for a story. Fair — and the answer is that we’re not throwing retention out. We’re refusing to stop at it. Retention is the price of entry; a thing nobody returns to clearly hasn’t found fit. But for intelligence, retention is the floor, not the ceiling. The ceiling is the decision. A team that returns daily and decides on gut has given you engagement, not fit, and the difference is the whole game.

There’s a deeper reason to insist on the harder metric: it’s the one that keeps a company honest. It is very easy to optimize usage — notifications, streaks, a slightly stickier interface — and feel like you’re winning while the intelligence changes nothing. Every lever that grows engagement is available to you, and most of them work on the metric without touching the value. You can manufacture a return visit. You cannot manufacture a changed decision; either the answer moved someone or it didn’t, and they know which. Holding yourself to “did a decision move” is uncomfortable precisely because it can’t be gamed. It’s the discipline that separates a durable intelligence company from one running on the fumes of early curiosity.

It also changes what you build. A team optimizing usage builds for the visit — the cleaner chart, the faster load, the reason to open the app again. A team optimizing for changed decisions builds for the moment after the answer: the next question it should provoke, the context that makes it trustworthy enough to act on, the path from “here’s what happened” to “here’s what to do.” Those are different products, and which metric you hold yourself to quietly decides which one you end up making.

This is what it means to deliver the conclusion, not the chart. The chart doesn’t disappear — it becomes the proof underneath the answer, the evidence trail you can audit. But it stops being the thing you hand over and call done. The product is the conclusion: the plain-language answer, the understanding it leaves behind, the decision it moves. You don’t get to count the visit, or the chart someone glanced at. You only get to count what changed because of it.

What we’re looking for

We’re at the stage where this stops being a thesis and becomes something you measure with real teams. So this is also an invitation. We’re looking for a few teams who want this bar held to their own data — who’d rather know whether an answer changed a decision than admire a usage chart. If “I can’t go back to dashboards” sounds less like a slogan and more like a bar you’d want to clear, that’s the conversation we want to have.

Key Takeaways

• For a tool, retention is the truth — its job is to be there when you return. For intelligence, the return visit stops being proof of anything.
• Usage is the vanity metric of the intelligence era: easy to grow, easy to celebrate, silent on whether a single decision changed.
• The real signals aren’t logins — the last answer generates the next question, a decision moves upstream of the product, and going back becomes intolerable.
• You can manufacture a return visit. You cannot manufacture a changed decision — which is exactly why the harder metric keeps a company honest.
• “Not the chart” doesn’t mean no charts — the chart becomes the proof beneath the answer. The conclusion is the product; the decision it moves is the bar.