There is a concept so foundational to scientific progress that it underpins virtually every credible advance in physics, medicine, and engineering, yet it rarely surfaces in conversations about marketing measurement.
That concept is falsifiability.
Coined by philosopher Karl Popper in the mid-twentieth century, falsifiability draws a sharp line between claims that can be tested and claims that merely sound persuasive. A hypothesis is scientific not because it's correct, but because it's structured in a way that allows reality to prove it wrong. When a claim can't be disproven, when no conceivable outcome would count as evidence against it, it isn't science. It's storytelling.
Marketing measurement has a falsifiability problem. And until the industry confronts it, billions of dollars will continue to be allocated on the basis of models that look rigorous but can't actually be held accountable.
How Science Actually Works
In fields governed by empirical discipline, progress follows a pattern. You state a hypothesis. Someone designs an experiment. The data either support the hypothesis or they don't. When the data contradict the hypothesis, the hypothesis is revised or discarded. Knowledge accumulates not because researchers are brilliant, but because the process is structurally honest. It forces you to be wrong, to learn from it, and to iterate.
This feedback loop is the engine of progress. Remove it, and you're left with persuasion masquerading as evidence.
The Unknowability Problem in Marketing
Marketing measurement should work the same way. In practice, it often doesn't, and the reason is more fundamental than most practitioners acknowledge.
When someone claims that "the incrementality of this channel is 5.9x," that number carries the weight of precision. It looks scientific. But verifying it is extraordinarily difficult, and true certainty is essentially out of reach. The actual incrementality of a marketing channel is not just unknown, it is, in a meaningful sense, unknowable. You cannot observe the counterfactual world in which you didn't spend that money. You can only estimate it.
Layer on top of that the familiar complications, flawed experiment designs, attribution time lags, hidden confounding variables, competitive dynamics, macroeconomic shifts, and the picture becomes even murkier. The number "5.9x" might be directionally useful, or it might be an artifact of the model's assumptions. Without a mechanism for testing that number against reality, there's no way to tell the difference.
This is the falsifiability gap in marketing measurement. Not a shortage of data or sophistication, but the absence of a structural mechanism for being proven wrong.
Why Traditional Statistical Measures Fall Short
The industry has developed a set of standard metrics meant to inspire confidence in models. Unfortunately, the most commonly cited ones don't address the core problem.
P-values are perhaps the most misunderstood statistic in applied analytics. In the context of a marketing mix model, a p-value tells you how likely you would be to observe the data given the model's assumptions. Crucially, it assumes the model itself is correct. A p-value cannot validate the model, it operates within the model's own framework. Using p-values to argue that a model is trustworthy is circular reasoning dressed in statistical notation.
R-squared and other in-sample goodness-of-fit metrics present a different but equally serious problem. A high R-squared tells you that the model fits the historical data well. It says nothing about whether the model reflects the actual causal structure of reality. It is straightforward to build models with near-perfect R-squared scores that are fundamentally wrong, models that have memorised the noise in past data rather than capturing the signal. This is the overfitting trap, and it is far more common than the industry likes to admit. Goodness of fit is not goodness of model. The distinction matters enormously.
These metrics are not useless, but they are insufficient. They measure internal consistency, not external validity. And external validity, the question of whether your model's claims hold up when tested against the real world, is the only thing that ultimately matters for decision-making.
The Case for Forecasting as the Accountability Mechanism
If the goal is to bring falsifiability into marketing measurement, the most practical path runs through forecasting.
A forecast is, by its nature, falsifiable. It takes the model's internal logic, translates it into a concrete prediction about the future, and then waits for reality to weigh in. The structure is simple and unambiguous: "If we spend X on this channel next quarter, we expect to generate Y in revenue." Then the quarter passes, and you compare the prediction to what actually happened.
The forecast either holds or it doesn't. There is no room for post hoc rationalisation, no way to retroactively adjust the goalposts. If the model predicted $4.2 million in revenue and the business delivered $2.8 million, something in the model is wrong. Maybe the channel coefficient was off. Maybe the model failed to account for a competitive launch or a shift in consumer behaviour. Whatever the cause, the discrepancy is a signal, a data point that forces the model to evolve.
This is the feedback loop that marketing measurement so often lacks. Forecasting introduces accountability that no amount of backtesting can replicate. Backtesting asks, "Can the model explain the past?" Forecasting asks, "Can the model predict the future?" Only one of those questions has teeth.
Building a Culture of Falsifiability
Adopting forecasting as a validation discipline requires more than a technical shift. It requires a cultural one.
Most organisations are not set up to reward models that are proven wrong. Analysts and vendors have strong incentives to present confident, stable numbers. Admitting that last quarter's model was off by 30% feels like failure. But in a falsifiability framework, that admission is the most valuable thing the model can produce. A model that is never wrong is not a good model, it is an untested one.
Practically, this means building forecasting into the regular operating rhythm. Before each planning cycle, the model should generate explicit, documented predictions. After the period closes, those predictions should be systematically compared to actuals. The gaps should be analysed, the model should be updated, and the next round of predictions should be generated. Over time, this cycle produces something that no single model output ever can: compounding confidence that the measurement system is actually reflecting reality.
It also means being transparent about uncertainty. A forecast should never be a single number. It should be a range, accompanied by the assumptions that drive it. When those assumptions are explicit, they become testable, and the organisation develops a shared language for discussing what the model does and doesn't know.
The Stakes of Getting This Right
The marketing industry spends enormous sums guided by measurement systems that have never been subjected to genuine falsification. Models are built, coefficients are estimated, budgets are reallocated, and the cycle repeats, often without anyone checking whether last quarter's predictions actually came true.
This is not a minor process gap. It is a structural vulnerability. Every dollar allocated based on an untested model is a dollar allocated on faith. And while faith has its place, it is not a sound basis for capital allocation at scale.
Falsifiability is not a philosophical nicety or an academic talking point. It is the mechanism that separates measurement systems you can trust from measurement systems that merely look trustworthy. It is the only reliable way to build confidence that your models are helping you make better decisions, and to catch it quickly when they aren't.
The question every marketing organisation should be asking is not "What does the model say?" It is "When was the last time we checked?"
