"It's been four weeks and the line is flat. Can we kill the test and move the budget to holiday?"
So you've got an upper-funnel test running on prospecting video. The power math said four weeks was enough. Four weeks are up, the holdout and exposed regions look about the same, and planning season is breathing down your neck.
Calling it now would be easy. It might also be wrong.
The math told you the shortest test that could spot a lift. It didn't tell you how long your customers take to buy.
Quick answer: A lift test (an incrementality test that compares a group that sees your ads with a holdout group that doesn't, by region or by user) has to fit two things. Power math sets the minimum length from your data's noise. Your customers' time to purchase sets the rest. For upper-funnel spend, run past the math's floor and add a post-period.
If you're still planning the test, the geo-test duration planner is the place to start.
Why do teams end lift tests early?
Pressure, mostly, plus a number that seems to back it up. Once the power math says the test is long enough and the early read looks flat, stopping feels safe. But that math only sets the shortest test that could spot a lift. It doesn't tell you how long your customers take to buy. For upper-funnel spend, it depends on that purchase time. Plan to run past the math's floor and add a post-period.
Holiday is coming and somebody wants that budget back. Planning season needs a number for the deck. The early read looks flat, so it feels safe to stop. And the power math already said the test was long enough, so who's going to argue?
That last one is the trap.
So what does the math actually promise?
What does the MDE math actually tell you?
But before you lean on that math, know what it measures. The minimum detectable effect (MDE) is the smallest lift your test can reliably tell apart from noise. It comes from how much your test and control groups already drift apart on a normal day, before any media changes.
Feed that noise into a power calculation and it tells you how long the test needs to run to detect a lift of that size, at the power you chose (often 80%). There's no universal MDE. It comes from your own sales history, brand by brand.
That's a useful number.
For how to size it on your own data, see the cost check in the feasibility guide or the geo-test calculator.
What does that math leave out?
Your customer.
The power math treats the lift as if it starts on day one. It measures the data's clock: how much noise you have and how long it takes for a lift to rise above it. It never asks whether your customers have had time to buy yet. A video ad seen today might not turn into an order for weeks.
It's like waiting on a delivery. The math tells you how long to watch the porch so you'd notice a package. It can't tell you when the package ships.
So which tests ship late?
Which tests need more time?
Any test where the customer is far from buying when they see the ad.
- Upper-funnel video and prospecting. People are meeting you for the first time.
- Cold, non-brand search. It's usually thought of as mid or upper funnel, so it needs time to bring in new buyers.
- High-AOV, long-consideration products. An upper-funnel brand video test for an expensive product with a long buying cycle may need much longer than six weeks.
Brand search sits at the other end. Someone searching your name may already be close to buying. (Planning a paid-search pause? See before you pause PPC, run this geo holdout.)
How do you know your own lag? I estimate it from click journeys, post-purchase surveys ("When did you first hear about us?") and past geo tests. A past upper-funnel test can show sales staying flat for weeks after launch, then moving. I walked through two public examples of how slowly this can show up in my piece on using AI to build an MMM.
The further your ad sits from the sale, the longer your test needs to run.
How do you set a length that fits the customer, not just the math?
But don't throw the power math out. It's your floor.
Then add your customers' clock on top. Take the purchase lag from your journeys, surveys and prior tests. Plan the treatment so it runs long enough to create new buyers and hold the lift long enough to beat the normal noise between groups. Then add a post-period (a cooldown after spend changes back) to catch orders that land late. The duration planner explains the observation period.
My rule of thumb is a 4-week minimum. Past that, "four to six weeks" is where the conversation starts, not a rule.
Example (hypothetical, simulated numbers): I'm planning a prospecting video geo test for a retailer. My four-week MDE range tops out around 2.5%, so I size to 2.5%: the power math says four weeks gives me 80% power at that size. But my post-purchase surveys and click journeys say most first-time buyers take about five weeks from first hearing about us to buying. So I plan a ten-week test plus a three-week post-period.
In weeks 1 to 4, weekly lift runs about 0.4% on average. That reads as flat. From week 5 to week 10, it runs about 5% in each of those weeks. By week 10, cumulative lift is about 3.2%, and the top of the ten-week MDE range has dropped to roughly 1.6%. If I'd stopped at week 4, I'd have called real lift a dud.
Numbers in this example are simulated for illustrative purposes. They are not a client result or a benchmark.
Which raises the next question: what if your calendar doesn't have ten spare weeks?
What about retail seasonality?
Careful with the calendar. Don't cut a test short to fit the holiday.
Plan around it.
And don't count on a normal week to tell you how Black Friday will behave. I'm not sure any test outside Black Friday can predict how high it can go. Whether another sale day, like Labor Day, acts like Black Friday varies a lot by brand. For some brands, Black Friday is a tentpole with nothing to compare it to. For others, it's a nothingburger year after year.
I'd rather finish a lift test before the season and use it to "calibrate" what last click is telling me. That gives me a last-click ROAS target adjusted by the test. Then I aim for that target during the hours of Black Friday. Could that be off? Yes. The lift on Black Friday may not match the lift in the test. But it's what we have, and math can only predict customer behavior so much.
If you only get one Black Friday a year, there's another option: run a geo test on Black Friday itself. This one is a scale-up test, not a holdout. The test regions get more spend, and the rest stay on plan. Keep promos the same in both groups. Then carry what you learn into next year. It lets you try a big swing with risk you can manage, then plan ahead. Some brands already do this.
Keep the promo calendar the same in your test and control regions, so a sale in one place doesn't look like lift. And write down any promo you can't avoid before it happens.
Is it OK to peek at results mid-test?
Picture this: week three, and the lift line ticks up. Everyone's excited. Week four, it dips back to zero, and now everyone wants to stop. Same test, two opposite calls, and neither read was scheduled.
Watch delivery and data quality every day. Make sure the holdout is actually dark and nothing else changed.
But save the read for the date you set before launch.
Write down the outcome, the regions and the read date in advance. The readout checklist in the feasibility guide covers what to commit to.
Check the plumbing daily. Read the result once, on the day you planned.
What if the test ended early and looks flat?
First, a caution: a flat result from a short test isn't proof the channel doesn't work.
A flat result is either underpowered, or, if you can show it was well powered, effectively negative. Here's the catch with slow-converting spend. A test can pass the power math, which assumes the lift starts on day one, and still end before most customers had time to buy. Then the lift inside the window is smaller than the one you powered for, so the test wasn't well powered for the question you asked. That's a design gap.
So don't go all in on the flat read, in either direction.
Re-run it with a design you can power for the full customer clock. That might mean a longer window, bigger regions, or a different question. If the channel feeds an MMM, see what an incrementality test can calibrate in an MMM before you plug in a short result.
FAQs
How long should an incrementality test run?
Long enough for two things. The power math has to say you can detect the lift you care about, and your customers need time to go from seeing the ad to buying. My rule of thumb is a 4-week minimum, and four to six weeks is where the conversation starts. Upper-funnel tests on long-cycle products may need much longer.
Is the minimum detectable effect enough to set test length?
No. The MDE tells you the smallest lift your data can detect and the shortest length that could detect it. It doesn't know how long your customers take to buy. Use it as the floor, then add your purchase lag and a post-period.
Why do upper-funnel lift tests need longer?
Because the people seeing the ad are further from buying. A prospecting video viewer may take weeks to place a first order. If the test ends before those orders land, the lift looks smaller than it is.
Can I stop a lift test early if the result looks flat?
It's risky. On upper-funnel spend, an early flat read can just mean the orders haven't arrived yet. Stop early only for a safety rule you agreed on before launch, like a holdout that's clearly hurting the business. Otherwise, read it on the planned date.
What's a post-period (cooldown), and do I need one?
A post-period is a stretch after the spend change ends where you keep measuring. It catches orders from people who saw the ads but bought later. For upper-funnel and long-consideration tests, plan one. The duration planner explains how it fits.
Should I run a lift test during the holidays?
You can, but don't let a normal test drift into Black Friday by accident. Holiday spikes add noise, and promos can differ by region. I'd rather finish a test before the season, use it to "calibrate" last-click numbers, and aim for that adjusted target on peak days. That target may be off on Black Friday, but it's what we have. If you only get one Black Friday a year, a geo test on the day itself, with more spend in the test regions, can teach you something for next year. Don't cut a test short just to fit the calendar.
Big picture
The math sets the floor.
Your customers set the length. When the pressure's on to call a test, ask one question first: have enough of our customers had time to buy? If the answer is no, the flat line isn't telling you much yet. Give it the weeks it needs, then read it on the day you planned!
Size the test for the data, then give it the time your customers need.
Need help? Speak with me here.