Incrementality testing for DTC & ecommerce

Causal evidence for media allocation

What Referral Analytics Help Prove Incrementality?

Referral analytics can identify a claim worth testing, such as whether referred visitors later buy or appear in a different mix of orders. They cannot prove incrementality without a credible comparison of what would have happened without the change in exposure.

Talk with Eric

Client-owned workDecision-first scopeUncertainty made visible

The question

Every channel can claim the sale. The business still has to allocate the next dollar.

Can referral or click analytics prove incrementality? Not on their own. They show which touches were near a sale, which is correlation, not cause. A holdout comparison, ideally a geo test designed before launch, shows what the media added. Plan it with the incrementality testing framework for DTC brands, and use the questions to ask an attribution vendor to check how a tool validates its numbers.

Testing is for a genuine dispute: two or three channels claiming the same sale, a view-through channel whose credit feels too far from the conversion, or a budget decision that click reporting cannot settle.

The useful question is not whether a dashboard is “right.” It is whether a strategy is adding to the business—and what changes if you spend less, hold steady, or scale it.

Keep or add?

Does this channel create incremental demand, or mostly record demand created somewhere else?

How much?

When a channel belongs in the mix, what can a levels test teach about marginal response and scale?

What changed?

Did the intervention cause a meaningful business outcome under the conditions tested?

A strong fit

Enough signal to make a decision.

Several meaningful sources of demand with competing claims on sales
A real allocation decision: keep, add, reduce, or scale a channel
About 12 months of outcome and media history, ideally with geographic detail
A decision owner willing to reconcile different measurement perspectives

Multi-million-dollar scale and multiple meaningful channels are useful heuristics—not universal gates. The test should follow the decision and the available signal.

Better to start elsewhere

When a test cannot answer usefully yet.

One paid channel drives nearly all meaningful demand, with no organic or other meaningful baseline
Too little history, variation, or volume for a credible comparison
A requirement that every report agree or that one dashboard must be the only answer

The next step may be cleaner data, a narrower attribution question, or waiting for a credible learning opportunity.

The comparison design

A geo holdout changes exposure. The real work is building the right comparison.

A treatment market is where media is introduced, paused, or changed. The question is what would have happened there otherwise.

Historical fit

Use prior sales to assess candidate markets before launching the test.

Treatment

Change media exposure in selected geographies while protecting the test conditions.

Comparison

Measure the outcome against a credible counterfactual—not a convenient dashboard read.

Decision

Translate the result, uncertainty, and next test into a bounded allocation move.

Matched markets

A stable historical cousin.

Compare the treatment geography with one or more places that behaved similarly over time. The fit must be stable, not merely similar on average after large swings.

Synthetic control

A weighted counterfactual.

Combine multiple geographies into a comparison designed to resemble the treatment market’s historical behavior. It can be powerful, but may require more of the country and limit concurrent tests.

The practical answer

Let the data choose.

Back-test both approaches against history. When volume and capacity allow, more than one credible comparison can increase confidence in a result.

The engagement

From a contested claim to a documented decision.

01
Discovery & feasibility
Clarify the decision, KPI, available data, current reports, seasonality, promotions, channel definitions, and the feasible learning path.
02
Test design
Validate candidate comparisons, define treatment and control conditions, assess what the test can detect, and establish the decision rule before launch.
03
Execution support
Check that media is running—or withheld—where intended. Identify exposure bleed, related-channel effects, and signals that need explanation while the test is in flight.
04
Analysis & handoff
Deliver a decision-ready readout, understandable charts, underlying documentation, uncertainty, and the next action or learning plan.

A measurement flywheel

Testing, MMM, and attribution should work together.

Attribution

Headlights on the road

Useful for the faster tactical view: creative, offer, audience, and observable near-term customer behavior.

Incrementality testing

Causal calibration

Establishes a bounded causal learning for a specific intervention under the conditions tested.

MMM

The portfolio view

Uses refreshed evidence to support broader allocation decisions across the media system.

A material test result can inform a deliberate MMM refresh. Attribution reports need not be rebuilt every time; the team can interpret their readings with a more calibrated understanding of where click credit is incomplete.

Before you sign

Choosing an incrementality testing vendor: what should you ask?

Disclosure: Brinker Analytics sells incrementality test design and readouts, so I have a commercial interest in how you answer this. The questions below work on any vendor, including me.

You can get a test run four ways: testing software, a vendor-run service, your own team, or a consultant. They look very different on a sales call. But the same questions sort all of them, because what you're really buying is the test design. That means the comparison, the power, the outcome, and what you'll do with each result.

Ask these before you sign.

What decision will this test settle? And what will we do if the lift comes in high, low or near zero? If nobody can answer that yet, the test isn't ready.
How will you build the comparison? Matched markets, a synthetic control, or a platform holdout? Ask to see the pre-period fit before anything launches. Vendors typically build around a house methodology and blend their models into a single answer; they rarely show you where methods disagree. So ask if they will. (More on matched markets vs. synthetic control.)
What can our sales volume detect, and over how long? Ask for the minimum detectable effect (MDE) and power over a proposed test length, from your own sales history. Then ask how that length fits your customers' buying cycle. I want several cycles in the window. Non-brand search usually needs longer than brand. My rule of thumb is a 4-week minimum, and shorter only if you'll accept a directional read and the risk of a null result. The geo-test calculator and duration planner give you a rough range first.
What outcome do you measure? Your backend orders, or the conversions a platform reports?
Can you test channels that leave little or no click trail? Think CTV, podcasts, radio or YouTube. Last click undervalues upper-funnel channels. They're by nature rarely the last thing someone clicks, and TV and radio never get a click at all.
In a holdout, which cell stays on? I call the cell that keeps running the treatment. Providers label this differently, so get it in writing before you read a report.
How do you report a null or inconclusive result? You want a range (an interval), not just one number. And you want a plain answer on whether the test was underpowered.
Can the results feed our MMM? Ask if you can export them and use them to calibrate the model. Here's what a test can and can't calibrate in an MMM.
Do we keep the data, the test designs and the code when the contract ends? Many platform plans are annual commitments. Ask now, not at renewal.
How is it priced? Per test, a subscription, tied to your ad spend, or a flat fee that doesn't rise with your revenue or ad spend? Ask, so you know what the next test will cost.

To plan the test itself, use the incrementality testing framework. Weighing whether to buy, build or hire? See why your attribution tools disagree. Buying attribution instead? Those vendors get a different list of questions.

If you'd like an independent read on a vendor's test design before you sign, talk through it with me.

Practical expectations

Rigor takes time. Fragmented data takes more.

The fastest work starts with accessible, consistently labeled data. Foundation work is sometimes the project—not an administrative prelude.

About one weekFocused measurement triage: decision, available evidence, likely method, and data gaps.

2–3 weeksFeasibility and test design once relevant data access is available.

Often 4–6 weeksIn-market runtime for sales or revenue; longer when the consideration cycle or intended effect is longer.

1–2 weeksAnalysis and handoff after the observation window closes.

Questions before committing

Can referral or click analytics prove incrementality?

Not on their own. Referral, pixel, and platform attribution show which touches were near a sale: correlation, not cause. To prove incrementality, compare places or people who saw the media with a holdout that did not, ideally with a geo test designed before launch. That test is also how you check whether an attribution model is accurate.

Do we need incrementality testing if platform reporting looks good?

Maybe. A platform can accurately report the conversions it observed while still being unable to answer how many would have happened without the advertising. Testing is most valuable when that difference would change a material allocation decision.

Can we test an established channel instead of turning it off?

Yes. When a channel clearly belongs in the mix, a levels test can be more useful: intentionally testing a lower, normal, or higher level of spend to learn about marginal response and possible diminishing returns.

Should we use matched markets or synthetic control?

Start with historical data. The right answer depends on which comparison is the more stable predictor for the proposed treatment markets and whether the business has enough volume and test capacity to support one or both approaches.

Will the test make every report agree?

No. Last click, MTA, MMM, and lift tests observe different evidence and answer different questions. The benefit is a clearer explanation of the disagreement and a more defensible decision—not a forced single number.

What should I ask an incrementality testing vendor?

Ask about the test design first. What decision will it settle? How will they build the comparison, and will they show the pre-period fit? What MDE and power can your sales support, over how long? Which outcome do they measure, how do they report a null result, and do you keep the data and designs?

A practical first conversation

Resolve the next media-allocation question with evidence you can explain.

Bring the channels, reports, and decision in front of you. We’ll determine whether a readiness diagnostic, a scoped incrementality test, or another measurement step is the practical next move.

Use the Geo Test Signal Planner first

Talk with Eric about incrementality testing