An incrementality test should not begin with “Does this channel work?” It should begin with the budget decision the business is willing to make, the smallest credible change that can create signal, and the comparison that can show what would have happened otherwise.
Download the one-page Incrementality Testing Framework handout for a planning meeting.
What is the best incrementality testing framework for DTC brands?
The best framework is the plan your team agrees on before the test starts, built around one budget decision. First, pick a change big enough to detect but small enough to afford. Then choose the outcome closest to sales that has enough volume. Back-test the comparison on your own history before media changes. Set safety rules for when to stop. Read the result as an estimate with a range, tied to the decision. Then act and plan the next test. If you want a second pair of eyes on your plan, talk it through with me.
1. Confirm that the intervention is proportionate
A test needs enough outcome volume and enough difference between treatment and comparison to produce a useful read. That does not mean a small business should double its spend just to get mathematical clarity. If the only way to create signal is a business-sized gamble, try the increase as a growth decision or keep using narrower experiments until the business can test a more modest change.
A practical first check is whether a roughly 10–20% change in exposure can still produce enough orders or another relevant outcome to learn from. The actual feasibility assessment uses the historical series, ordinary variation, candidate markets, and the decision at stake.
There is no universal minimum detectable effect. Your own history sets it. The day-to-day noise in sales, leads, or whichever outcome you measure determines how large a lift a test can reliably see, and that varies brand by brand. A lift threshold borrowed from another company's test tells you very little about yours. For a directional read before detailed feasibility work, try the Geo Test Signal Planner and the geo-test duration planner.
2. State the allocation decision
“Does Meta work?” is too broad. A decision-ready question might be: “At what level does another dollar in Meta stop being the best next use of the budget?” A holdout, normal-spend, and spend-up design can create points on a response curve. The aim is to understand marginal return, not simply confirm that a historically useful channel exists.
The response curve applies to the conditions tested. Creative, offer, audience, season, and competition can change it. Use it to decide the next move, then keep learning.
3. Choose the closest feasible outcome
Choose the measure that is closest to business success, stable enough to project forward, and plentiful enough to create a clear signal. Revenue often fits. But if a promotion changes average order value sharply, orders may be a better primary outcome. Branded search or another nearer signal can be useful when sales volume is too thin, but only if history supports its relationship to the business result.
Clicks are easy to obtain and easy to misread. A proxy should not become the goal merely because it is abundant.
4. Validate the counterfactual before launch
The comparison must predict the untreated path. States can be simple to assign, but there are only 50 of them. DMAs provide more candidate units and can support a synthetic control: a weighted combination of other markets that reproduces the treatment group's historical movement.
Do not accept a comparison because it looks reasonable on one chart. Build it on one historical period, then check whether it predicts a separate, untouched period. That out-of-sample check is the practical proof that the counterfactual deserves trust.
Then run the pre-test baseline checks. Measure how well the controls predicted the test markets in the pre-period, using measures such as MAPE and the correlation between test and control. Run placebo checks and apply your control-selection rules before any media changes. A comparison that cannot hold up on history has not earned a place in the live test.
For the comparison-design details, read matched markets versus synthetic control. For the full pre-launch checklist, read Before You Call a Geo Test Feasible.
5. Set three kinds of safety rule
Validity: stop or redesign if the treatment is not actually happening. Business survival: restore the usual setup when the company cannot afford the loss required by the holdout. Decision speed: an owner may accept a less certain, early lower-funnel read to move faster, but that is an explicit business tradeoff, not a statistically final result.
Where fidelity matters and the business can absorb the risk, keep the planned window. Use the geo-test duration planner to frame the active period and cooldown before detailed feasibility work, or use the Geo Test Signal Planner for a directional read of how volume, variation, and consideration time change the planning signal.
6. Read the result as a decision
Show the estimate and its range, not only one score. If both ends of the plausible range support the same action, the owner may not need to split hairs. Where practical, compare more than one credible view: a matched-market comparison, a synthetic control, and a wider national trend. Agreement does not manufacture proof, but it can show whether the learning survives reasonable comparison choices.
Review orders, revenue, media cost, and the action rule together. Explain how a change in one channel may alter what another channel captures. That is one reason last-click reporting alone cannot settle a serious allocation question.
An inconclusive result usually means one of two things. The test was underpowered, or, if you can show it was well-powered, it is effectively a negative, because it is not a clear win. Either way, the next move is a test you can confidently power. That may be the same hypothesis with a different cell design, or a different hypothesis.
How do you calculate lift and iROAS?
Step 6 says to read the result as a decision. So you need two numbers first: the lift, and what that lift was worth per dollar.
Lift is how far the treatment group moved compared to what your control says it would have done anyway. That “would have done anyway” number is the counterfactual (your best estimate of sales with no change).
- Incremental sales = actual sales in treatment − counterfactual sales in treatment.
- Lift % = incremental sales ÷ counterfactual sales.
- iROAS (incremental return on ad spend) = incremental revenue ÷ the spend you changed.
For a holdout, flip the direction. You turned spend off, so the lift shows up as a drop. Here the counterfactual is what the holdout geos would have sold with spend still on. Incremental sales are that number minus actual sales, and the spend is what you paused.
What's a good iROAS?
State it on revenue, then compare it to your breakeven, which comes from margin: breakeven iROAS = 1 ÷ contribution margin (after product, shipping, fees and other variable costs). At a 55% contribution margin, that's about 1.82x. This only works when iROAS is stated on revenue.
If your target iROAS sits below breakeven, a “win” still loses money. Rethink the target before you launch.
Example: my target for this channel is 2.5x iROAS. Over a 12-week test, the control says treatment geos would have done $400,000. They actually did $440,000. That's $40,000 in incremental revenue, a 10% lift. We added $16,000 in spend, so iROAS is $40,000 ÷ $16,000 = 2.5x. Against a 1.82x breakeven (55% contribution margin), that clears.
Numbers in this example are simulated for illustrative purposes. They are not a client result or a benchmark.
But don't stop at one number: show the range around it. If both ends of the range point to the same action, act.
How close is close enough? Write that down before launch. One way I read it: within about ±20% of the iROAS you expected, act on it. That band can widen or narrow with your appetite for change and your volume. Far above or far below, make a smaller change and retest, or design a different test.
One caution. If the channel you tested pushed up sales that another channel's spend also chased (for example, a YouTube test could shift spend or results in another channel, like Facebook), the sales lift alone doesn't tell you the full iROAS. Count the spend that moved with it.
To plan the spend you'll need before launch, use the geo-test feasibility checks.
7. Act, then design the next test
Agree before launch: if the result supports the hypothesis, what changes? A positive holdout can justify national deployment, followed by a spend-up test that asks how far to scale. When a test produces a credible hard learning, it can also anchor the wider portfolio discussion in an MMM. It is an input with conditions and limits, not a universal rule for every channel.
Example: applying the framework to a YouTube test
YouTube is a hard case for click-based reporting. Someone watches on a television, then searches for the brand later, and the click trail never connects the two. That makes step 3 the deciding step: the outcome may show up in branded search, ecommerce, marketplaces, or retail rather than a tracked click, and step 5's window has to allow for that delay.
Use the framework in a planning meeting
Bring the decision, primary outcome, expected treatment, candidate comparison, business-loss limit, and action rule. If any of those items are absent, the next useful step may be reporting work, a narrower experiment, or a different allocation question.
Eric has run more than 70 incrementality tests over about three years: geo-level and user-level lift tests across several channels, including tests on different channels run at the same time.
Download the one-page framework
Questions about incrementality testing frameworks
What is an incrementality testing framework?
It is the plan you agree before a test starts, not the analysis you run afterward. It names the budget decision, a change you can afford that can still be detected, the outcome, a comparison back-tested on history, the stopping rules, and what you will do with each result. The seven steps above are one version of that plan.
How long should an incrementality test run?
Four to six weeks is a reasonable place to start the conversation, not a rule. The power math needed to confirm a non-null result sets the length, adjusted for how the tactic is expected to drive action. An upper-funnel brand video for a high-priced product with a long consideration cycle may need much longer than six weeks. The geo-test duration planner gives a directional read for your inputs.
How big a lift can my test detect?
There is no universal answer. Your own history sets it: the daily noise in the outcome you measure, and your volume, determine how large a lift a test can reliably see. Check how well candidate controls predicted the test markets before launch, using measures such as MAPE and correlation, and use the Geo Test Signal Planner for a directional read.
Should I use a geo test or a platform (user-level) lift study?
When you can, run both at the same time and compare. If you can't, or they disagree, rely on the geo-level test; a user-level lift study is mainly a convenience.
In one case, a Facebook geo test and a Facebook user-level lift test run at the same time returned about the same result. The brand then used user-level lift for Facebook going forward, which freed its geo-test roadmap for other channels.
What if the test result is inconclusive?
Either the test was underpowered or, if you can show it was well-powered, it is effectively a negative, because it is not a clear win. The next step is a test you can confidently power: the same hypothesis with a different cell design, or a different hypothesis.
How does a test result work with an MMM?
A credible test result can calibrate an MMM for the channel and conditions it covered, and an MMM can point to the next question worth testing. The result carries its conditions with it, so it should not become a rule for every channel. Read what an incrementality test can calibrate in an MMM for how much weight to give it.
Is this the same as the incrementality testing service?
No. This is the planning framework. For help scoping a test, see Incrementality Testing for DTC Brands.