Your pilot proves nothing

A client told me recently that their agent pilot had been a success. Tickets were down eleven per cent since launch.

I asked what else had changed in those four months. After some thought: a seasonal lull, two experienced people joining the team, a knowledge base refresh, and a separate initiative to close duplicate tickets at intake.

So the eleven per cent belongs to five candidates and we have no way of allocating it. That is not a business case. It is a coincidence they paid for.

The ladder is not a metaphor, it is a hierarchy

Judea Pearl, who won the Turing Award in 2011 for this body of work, describes three rungs of causal reasoning, and the distinction between them is the most useful thing an executive can learn about AI this year.

Rung one is association. What does the data show? Tickets went down after we launched the agent. This is seeing. It is what correlation gives you and it is all that most dashboards can give you.

Rung two is intervention. What happens if we act? If we route these ten ticket types to an agent, cost per ticket moves by this much. This is doing, and it requires an experiment.

Rung three is the counterfactual. What would have happened if we had not? Without the agent we would have spent this much. That difference is the saving. This is imagining, and it is the only rung on which a business case can actually stand.

Almost every AI steering committee pack I have been shown was written on rung one.

Rung one has been known to be insufficient since 1739

David Hume made the point that has never been successfully answered. We never observe causation. We observe one billiard ball strike another and the second move away. What arrives through the senses is constant conjunction, this and then that, and the mind supplies the causal link because it is useful to do so, not because it was perceived.

Correlation is what the world gives you for free. Causation is something you have to construct.

The method for constructing it is not new either. John Stuart Mill published his Method of Difference in 1843: if a situation in which a phenomenon occurs and one in which it does not differ in only one circumstance, that circumstance is the cause.

That is a control group. It is a hundred and eighty years old, and a great deal of eight-figure AI expenditure is currently being approved on evidence that would have failed it.

Ronald Fisher gave the method its teeth in 1935 with randomisation. The famous opening of The Design of Experiments has a colleague claiming she can taste whether milk went into the cup before or after the tea. Fisher's response was not to argue but to design: eight cups, four each way, randomised order. Randomisation is what licenses the causal claim.

And John Snow showed in 1854 what rung three looks like in the field. Everyone remembers the Broad Street pump. The better study is the one where he noticed two water companies serving intermingled households on the same South London streets, one drawing water downstream of the sewage outflows and one that had moved its intake upstream. Same neighbourhoods, same class, same air, one variable different. He had no germ theory. He had a counterfactual, and it was enough to move a pump handle and save a city.

You have more data than Snow ever dreamed of, and no counterfactual at all.

The one question that exposes it

Ask your team this: what did it cost us to process one ticket, one claim, one invoice in January, before the agent existed?

If the answer takes more than a day to produce, you do not have a baseline. Without a baseline there is no counterfactual, and without a counterfactual there is no defence when the line item comes up at budget review. Not a weak defence. None.

Watch for the aggregate answer, because it is the most common evasion and it sounds credible. Total departmental spend is not a baseline. If I cannot divide it by the number of things you did, it cannot tell me whether you got better at doing them.

Why capable people keep skipping this

Two reasons, and neither is stupidity.

The first is that the baseline is boring and the demo is thrilling. Capturing cost per unit takes about a week of unglamorous work by someone who could otherwise be building. Nobody has ever been promoted for it.

The second is Michael Polanyi's observation from 1966 that we can know more than we can tell. Your best claims assessor cannot fully articulate how she spots a suspicious file. That tacit knowledge is exactly what the agent needs to replicate and exactly what nobody wrote down, which means the process was never documented well enough to be measured in the first place. Process debt is tacit knowledge debt, and the baseline exercise is where you discover you have it.

Evidence without an owner is just an opinion with numbers on it

There is a harder lesson underneath the technical one, and Ignaz Semmelweis is the person to learn it from.

Vienna General Hospital, 1847. He notices that the maternity ward staffed by doctors has a maternal death rate several times higher than the one staffed by midwives. He does what nobody else does: he measures both, month by month. He works out that the doctors are arriving at deliveries straight from the dissection room. He orders chlorine handwashing, and the mortality in the doctors' ward collapses to match the midwives'.

He had the baseline. He had the intervention. He had the counterfactual sitting in the ward next door. The profession rejected him anyway. He was pushed out and died in an asylum in 1865, before Pasteur and Lister made his conclusion respectable.

Having the evidence is necessary and it is not sufficient. Someone senior has to own it, which is why the first thing I ask organisations to do is name a human being against every agent running in production. Not a team. A name.

What to do this quarter

Capture the baseline before the next pilot starts. Cost per unit for the workflow you are about to touch. One week. Do it before, because after is not a baseline, it is a memory.

Find a control. A comparable team, region or ticket type that does not get the agent. Snow's two water companies were handed to him by accident. You can arrange yours on purpose.

Write the counterfactual sentence before you build. "Without this agent, we would have spent X." If you cannot complete that sentence with a number and a method, you are not building a business case. You are building a demonstration, and you are paying for it.

A pilot without a counterfactual is not evidence. It is a coincidence you paid for.