What is a geo test power analysis?
A geo test power analysis tells you, before launch, how likely your test is to see the lift you're planning for. Aim for 80% power at 90% confidence, one-sided. If power is low, run longer first. Then spend more in the test cell. Then make the cell bigger. Don't tiptoe in.
"Power" is the chance your test catches a real lift of the size you planned. At 80%, a real lift of that size shows up four times out of five.
The "minimum detectable effect" (MDE) is the smallest lift the test can reliably see at your chosen power (here 80%). Anything smaller just blends into the noise.
Here's the real question behind the math. Will you spend hard enough, run long enough, and believe in the plan enough to get a lift the test can see?
What's the biggest mistake teams make?
Hedging.
A team likes an idea, but not quite enough. They put in a little spend. They run it for a short stretch. Then the readout comes back and everyone shrugs.
"We couldn't tell."
What they mean is, we were afraid to try harder. That test cost money and taught nothing.
If you're going to test a strategy, swing for the fences.
Some tests are bold by nature. A holdout, where you turn a channel off in some regions, is usually a big enough change to see. (Here's a search-pause example of that kind of holdout.)
Other tests are subtle. Say you want to compare a 1-day click attribution setting with a 7-day one. That change may be too small for any geo test to pick up. Be honest about which kind you're running.
How does the math work? (an example)
The math here is light. It's only shown so you can see where the numbers come from.
Illustrative example, sample data, not a real client. All numbers below are simulated for illustration. The standard errors are assumed, not measured.
Two terms first. "Standard error" (SE) is how much your result would bounce around from pure noise. "iROAS" (incremental return on ad spend) is the extra sales you get per dollar of test spend.
The formulas
- MDE ≈ (1.28 + 0.84) × SE. The 1.28 is for 90% one-sided confidence. The 0.84 is for 80% power.
- Power at your planned lift ≈ Φ(planned lift ÷ SE − 1.28). Φ just turns a score into a percent.
- Spend needed to reach the MDE = MDE × baseline test-cell sales ÷ target iROAS.
Example: Say I want a channel to return 2.0x iROAS. I plan a 4-week test. My test cell would sell $400,000 in that window without the extra spend. I put in $8,000. At 2.0x, that should add $16,000 in sales, a 4% lift.
| Plan | Window | Baseline test-cell sales | Test spend | Target iROAS | Planned lift | SE (assumed) | MDE | Power |
|---|---|---|---|---|---|---|---|---|
| A: tiptoe | 4 weeks | $400,000 | $8,000 | 2.0 | 4.0% ($16,000) | 2.5% | 5.3% | ~63% |
| B: run longer | 8 weeks | $800,000 | $16,000 (same weekly) | 2.0 | 4.0% ($32,000) | 1.8% | 3.8% | ~83% |
Sample data, simulated for illustration.
Plan A. The MDE is 2.12 × 2.5% = 5.3%, so roughly 5–6%. My planned lift is only 4%. Power is Φ(4.0 ÷ 2.5 − 1.28) = Φ(0.32), or about 63%.
To reach the MDE, I'd need 0.053 × $400,000 ÷ 2 = $10,600 in spend. I planned $8,000. Too light.
Plan B. Same weekly spend, twice as long. More data shrinks the noise, so SE drops to 1.8%. The MDE is 2.12 × 1.8% = about 3.8%, so roughly 3.5–4%. Power is Φ(4.0 ÷ 1.8 − 1.28) = Φ(0.94), or about 83%. That passes 80%.
Spend to reach the MDE is 0.0382 × $800,000 ÷ 2 ≈ $15,300 (exact MDE 3.82%). I planned $16,000. Enough.
The takeaway is simple. Nothing changed except time, and now the test can see the lift.
If 8 weeks isn't possible, raise the 4-week spend to about $10,600 or more (lever 2), then grow the cell (lever 3).
One caveat on those ranges. The SE is an assumption. The ranges show what happens to the MDE if the SE is off by about 0.25 points either way.
Want to run this on your own numbers? Use the geo test calculator.
What should you change first when power is low?
Pull these levers in this order.
- Run longer first. Customer journeys take time. If people take three months to buy, a one-week spend spike won't show up. The geo test duration planner helps you pick a length.
- Then raise spend in the test cell. Scale it to the cell's share. For example, if the cell is 20% of the country, its spend should match that 20% share at the higher level you're testing.
- Then make the cell bigger. A bigger cell gives you more data.
In real life you may need both two and three.
One more thing. I like to keep the treatment cell at 25% or less. That's my personal taste, not a technical rule. In a holdout, the treatment is the cell that stays on, and the cell you switch off is the control.
Is 60–70% power ever OK?
Sometimes. But only if you get something real in return.
A good trade looks like this: a rapid 2-week directional test. It's quick, and it lets the brand take more swings with fewer big commitments. You treat the result as a hint, not a verdict.
A bad trade looks like this: a 6-to-8-week test at 60% power. I'd hate that one. You pay for a long test and get odds not much better than a coin flip.
Lower power is fine when it buys you speed. It isn't fine when it buys you nothing.
When shouldn't you run the test at all?
Skip the geo test when:
- Your portfolio is simple. If you run one channel, an A/B test may answer the question better. See the incrementality testing framework for how to pick.
- No one plans to act on it. If a win wouldn't change anything, such as rolling it out nationwide, don't spend the money.
- You don't have the time. You need at least a 3-to-4-week window.
- A big promo is in the window. Black Friday and similar spikes usually swamp the signal. There are some exceptions.
For the full feasibility checks, see before you call a geo test feasible.
What does a well-powered flat result mean?
It means the idea was not a home run.
A well-powered test that "couldn't prove a result" still tells you something. If the lift were at least the size you planned for, you'd likely have seen it. Take that and move to your next idea.
But what if you still believe in it?
Then retry with a safety net. Raise the power with a longer run, more spend, or a bigger cell. And use more than one scaled control group. (A scaled control is a group of regions multiplied to match the test cell before launch.)
Only do that if your pre-test health checks allow it. That means low "MAPE" (mean absolute percentage error, or how far off the control's fit is) and high correlation between test and control. Here's how to run those comparison checks.
Why does this matter for channel budgets?
Last click undervalues upper-funnel channels. Their effect often shows up later as other channels' clicks. A well-powered geo test is how you see what those channels really add, and it's one more read to set beside your platform reports and an MMM. No model owns the truth. A good test just gives you a better one to weigh.
Want a second set of eyes on your power plan? Talk through a test design.
FAQ
What power should a geo test have?
Aim for 80% power at 90% confidence, one-sided.
What if my power score is too low?
Run longer first. Then raise spend in the test cell. Then make the cell bigger.
Is a 2-week test at 60% power useful?
Yes, as a quick directional swing. Don't treat it as a final answer.
My test came back flat. Was it a waste?
Not if it was well powered. It tells you the idea wasn't a home run, so you can move on.
How big should the treatment cell be?
I prefer 25% or less. That's my personal taste, not a rule.
Want help sizing your next test? See if we're a fit.