Was it the ad, or was it August?

A business owner changes something — new ads, a rebuilt website, a different price. A month later the numbers are better. The conclusion writes itself: it worked.

That’s a before-and-after comparison. It is the most common piece of analysis in business, it feels like evidence, and it is the conclusion most likely to be quietly wrong — because the world did not hold still while you made your change. School let out. A competitor’s best crew quit. Hurricane season started. Google reshuffled something in a data center you’ll never see. Any one of those can move your numbers in exactly the way your change was supposed to.

So “did it work?” turns out to be a question about a world you never got to see: the one where you changed nothing and August happened anyway. You can’t observe that world. You have to construct it, and constructing it honestly is a skill — one I spent a doctorate on.

My dissertation measured how shocks to a neighborhood — a school, a crime pattern, new infrastructure — move property values. Nobody lets you run a neighborhood twice to see what would have happened without the shock. The whole discipline is finding a comparison that stands in for the world you’re missing: a neighborhood that matched in every way except the one thing that changed, so the difference between their paths is the effect, isolated. Economists call the workhorse version difference-in-differences. The name is ugly. The idea is just honest comparison.

Here’s the business translation. An air-conditioning company relaunches its website in July. In August, the phone rings constantly. The owner credits the new site — and maybe he’s right. But this is Florida, and it’s August. The month itself is the best salesman that company has. From before-and-after alone, a great website and a hot summer are indistinguishable.

What breaks the tie is looking where the change couldn’t reach. If the new site is doing the work, the gains should be shaped like the site: the pages that changed pull ahead, the sources the site improved grow faster than the ones it never touched. If instead everything rose by the same tide — old pages and new, every source at once — you’re looking at August, not the website. The comparison you need is usually sitting inside your own numbers, unexamined.

There’s a proportionality rule hiding here, and it’s the part I wish more analytical people respected: not every decision deserves this care. A cheap, reversible change can survive sloppy inference — try it, keep it, move on. But expensive, repeated decisions — doubling the ad budget, rebuilding the site again next year, hiring against a growth number — compound. Get the cause wrong once and you don’t just waste the first dollar; you commit the next ones to the same mistake. Rigor should be spent where the money is.

This is what I actually mean when I say Pruvara improves things from evidence. The measurement is continuous, but measurement is the easy half. The discipline is refusing to credit a change until the shape of the gains matches the shape of the change — asking, every time, what would have happened anyway. It’s a less flattering way to read your own numbers. It’s also the only reading that deserves your next dollar.

Every dashboard in the world answers “what happened.” The expensive question is “what would have happened anyway” — and it’s the one question the dashboard can’t answer on its own.