"Should we build an MMM, or just run more tests?"
It's a fair question. And it usually comes up after the platform reports stop adding up.
Here's the short version.
Quick answer: An MMM (marketing mix model) estimates how each channel contributes to sales across your whole budget, using your history. An incrementality test measures what one change added, against a controlled comparison. They answer different questions. They work best together: tests check the model, and the model points to the next test worth running.
What does an MMM tell you that a test can't?
An MMM looks at the whole budget at once. It's a statistical model of how spend in each channel moved sales over time. It also accounts for things like seasonality, promotions and demand.
That makes it a portfolio tool. When several channels compete for the same dollars, an MMM puts them in one view. It can include media people never click, like TV or podcasts. It can also suggest where the next budget move might pay off.
A test can't do that. It reads one change, in one period, at one spend level.
But an MMM needs something to learn from. My working starting point is at least 12 months of consistent sales and spend history, with real changes in spend. If a channel never moved on its own, the model has a hard time pulling it apart from everything else.
An MMM is a map of the whole budget. It's only as good as the history behind it.
What does an incrementality test tell you that an MMM can't?
A test answers a cause question. Did this change add sales?
You make a defined change, like turning a channel off in some regions. Then you compare sales against a group that didn't get the change. The gap is your estimate of what the change did. A geo test is the common way to do this for a DTC brand, because it reads total sales in your own order record.
An MMM works from patterns in history. A good test creates the variation on purpose, so it can stand as stronger evidence for the channel it tested.
The catch? The result carries its conditions with it. It describes that channel, at that spend, in that season. One test is one data point, not a rule for every channel.
When should you run a test instead of building an MMM?
Start with the decision on your desk.
Run a test first when one channel's budget call is the big question. Maybe you're thinking of pausing it. Maybe the platform claims more sales than you think it drives. A test can answer that without a model.
A test also makes sense when you don't have the history yet. If you haven't spent at a steady level for long, an MMM has little to learn from. A test can still give you a read on your biggest channel.
Build an MMM when several real channels compete for budget and no single test can split it for you. Do you need an MMM yet? walks through the checklist.
How do MMM and incrementality tests work together?
A credible test can tell an MMM what's plausible for the channel it covered. In many models, that means it informs the range of effects the model will accept for that channel. The MMM then reads the rest of the budget. And where the model is least sure, that's a good place for the next test. Here's what an incrementality test can and can't do inside an MMM, including which tests deserve the weight.
What about MTA?
MTA (multi-touch attribution) is a third tool, with a third job. It shows which tracked touches came before a sale and splits the credit. That's useful for seeing journeys. But it can't show what the media caused. Last click undervalues upper-funnel channels, and MTA has the same blind spot for any touch it can't track. Here's why MTA and an MMM often disagree.
MMM vs incrementality testing at a glance
| MMM | Incrementality test | |
|---|---|---|
| Question it answers | How each channel contributes across the whole budget | What one change added |
| Data it needs | Sales and spend history across channels, with real changes in spend | Sales in test and comparison regions or groups, plus a pre-test history to check the comparison |
| Time to a read | Once the history is in place and the model is built and checked | The length of the test. My rule of thumb is a 4-week minimum |
| What it can't see | Order-level detail, and anything the history never varied | Every channel at once, and conditions outside the test |
Neither one owns the truth. No model does.
That's the point of using both. Each one checks the other.
If you're not sure which comes first for your brand, talk through your budget question with me. Or start with the incrementality testing framework to plan a test.
FAQs
Is MMM or incrementality testing better?
Neither. They answer different questions. An MMM splits the whole budget using history. A test measures what one change added. Many brands with several channels in play use both, with tests checking the model.
Can a small DTC brand run both?
Often a test comes first. An MMM needs enough history and spend changes to learn from. A test can still answer your biggest channel question now. Add an MMM once several channels compete for budget and the history is there.
How many tests does an MMM need?
There's no set number. Start with the channels behind your biggest budget questions. See how many tests you need.