"Do we even need Google Ads? Let's just turn it off and see what happens."
So you've got a paid search line that's one of the biggest in the budget. Google Ads says it's working. Your attribution report mostly agrees. But leadership knows the platform is grading its own homework, and now somebody wants to pull the plug to find out.
Fair question. Bad test.
Quick answer: Don't switch paid search off everywhere. Pause it in a set of holdout regions, keep it running in matched treatment regions, and compare sales over a window you plan in advance. Then keep, trim or cut the spend, using a rule you agreed on before the test started.
That's a geo holdout (a test where ads go dark in some places and stay on in others). It answers "is Google Ads worth it?" with your own sales, not the platform's report.
Pausing search everywhere tells you almost nothing. Pausing it in the right places, with the decision written down first, tells you what it's worth.
Why not just turn Google Ads off and watch sales?
Picture this: you pause all paid search on the first of the month. Sales dip 4% over the next three weeks. But a promo ended that same week, Meta spend shifted, and it's the slow season anyway. Which one caused the dip?
You can't tell.
When everything moves at once, a national pause gives you a before-and-after with no clean comparison. Organic search may also pick up some of the clicks your ads used to get. So the drop you see could be bigger or smaller than what the ads were really driving.
This isn't a new worry. In a well-known set of experiments at eBay, economists found that brand-keyword ads had no measurable short-term benefit, and that paid search returns overall were a fraction of what non-experimental estimates suggested (Blake, Nosko and Tadelis, NBER working paper 20171). That's one company, tested years ago. Your brand may be very different. Which is exactly why you test yours.
What decision are you actually making?
Write it down first.
One sentence each. Which campaigns are in the test: brand, non-brand, Shopping or Performance Max? What result means keep, what means trim, and what means cut? And who signs off on the call? Clear that last part with the decision-maker before anything goes dark.
"See if search is incremental" isn't a decision. "Keep, trim or cut non-brand search next quarter" is.
Should you pause brand and non-brand together?
One caution before you pick campaigns: pausing brand and non-brand at once muddies the read. Test one thing at a time.
Brand and non-brand do different jobs. Brand search often catches people who already know you and are on their way to buy. Non-brand is more likely to find people who've never heard of you. Pause both at once and you can't tell which one the result belongs to.
So pick one.
Then list what stays on: Meta, email, promos, and anything else running in those regions. Freeze planned changes where you can. Once the list is set, move on to the map.
How do you pick holdout and treatment regions?
Use regions your data can see.
That usually means states or DMAs (designated market areas, the TV market regions Nielsen defines). Check that your analytics can report sales at that level before you pick. Pick them by matching or at random, based on your own sales history. Don't just grab a couple of small test markets because they feel safe. And keep giant outlier markets, the ones that behave like nobody else, out of the holdout.
A quick note on labels. In a holdout, I call the regions where search stays on the treatment, and the regions that go dark the holdout. Providers label this differently, so check which is which before you read anyone's report.
Before you commit, check that the holdout and treatment regions have moved together in the past. The geo-test feasibility checks walk through how, with free code on GitHub. For building the comparison itself, see matched markets vs. synthetic control.
How big does the holdout need to be?
Add up two shares for the holdout regions: how much of your search spend goes dark, and how much of your sales sits there.
Too small a holdout can't show anything. The lift you lose gets buried in normal day-to-day noise. And there's no universal number for "big enough." The smallest change you can detect, the minimum detectable effect (MDE), comes from your own sales history.
The geo-test calculator gives you a quick range. The cost check in the feasibility guide shows how to size a holdout against your own noise.
If the spend you switch off can't move sales more than your normal noise does, the test can't answer the question.
How long should search stay off?
Think of skipping water on a garden. Skip one day and everything looks fine. Skip a few weeks and you find out which plants needed it. One week dark doesn't tell you much either.
Start with the math. Calculate the MDE and power over a proposed test length, using your own sales history. Then check that length against your customers' clock.
That clock is the consideration cycle: the time from seeing an ad to buying. I want several cycles of data in the test. Non-brand search usually has a longer path to conversion than brand search, so it usually needs a longer holdout.
In my view, too few providers treat the customer cycle as a variable at all. It belongs in the plan right next to power.
Example (illustrative): Say brand search takes about 2 days from search to purchase. Several cycles is about 14 days. But the MDE math may need 4 weeks to reach acceptable power, so power sets the length. Now say non-brand search takes about 14 days. A few cycles is about 6 weeks, and that may already satisfy MDE and power. Here the customer sets the length.
These cycle lengths are illustrative. They are not benchmarks.
My rule of thumb is 4 weeks minimum. I'd only go shorter for a brand with high risk tolerance, one that accepts directional data and the chance of a null result in exchange for speed.
Plan a short observation period after you turn search back on, too. Some sales land late. The geo-test duration planner helps you rough out the window. And ending a lift test too early is its own trap.
How do you set up the pause so it's clean?
But a pause is only as clean as its settings.
In Google Ads, add the holdout regions as excluded locations. Do it in every campaign that could serve search ads there, including Shopping and Performance Max. They each have their own location settings, so don't assume one change covers all of them. If you run Microsoft Advertising, exclude the same regions there. Microsoft applies ad-group location settings instead of the campaign's when an ad group has its own, so check both levels.
Then check who you're targeting. Google's default reaches people in your locations or interested in them (it calls this "Presence or Interest"). Its "Presence" option limits ads to people in, or regularly in, those places. For exclusions, "Presence" is the only option on Search, Shopping and Performance Max.
Expect a little spillover. Google says it places people by IP address, device location and other signals, and that 100% accuracy isn't guaranteed. It may also show ads in nearby related areas. People travel. That's normal noise, as long as it's small and the same on both sides.
Keep a change log. Every setting, date and person who touched the account goes in it. Check the current setting names on Google's location targeting help the day you launch.
What should you watch while it runs?
Careful here. Watching delivery is good. Watching the result every morning is how tests get stopped early.
Each day, check that spend in the holdout really is near zero. Check that nothing else changed in those regions: no surprise promo, no new campaign, no budget shift. Note anything odd in the change log. Leave the sales comparison alone until the read date.
If you feel the urge to call it early, read why ending tests early goes wrong.
How do you read the result?
Picture this: the treatment regions, where search stayed on, tell you what the holdout regions would have sold with search still on. The holdout regions sold less. The gap is your incremental sales.
From there:
- Incremental sales = predicted sales in the holdout (from the treatment regions) minus actual sales.
- iROAS (incremental return on ad spend) = incremental revenue ÷ the search spend you paused.
- Breakeven iROAS = 1 ÷ contribution margin. At a 55% margin, that's about 1.82x. (This works when iROAS is measured in revenue.)
Show a range, not one number. Then compare it with what Google Ads reported for the same period.
Don't be surprised by a gap. In my experience, it varies wildly. Often, though, lower-funnel strategies swing higher in platform than in holdout, and upper-funnel strategies swing lower in platform than in holdout. (Why the numbers disagree at all is its own question.)
What if the result is flat? Then the test was either underpowered, or, if you can show it was well powered, the result is effectively negative. It's not a clear win either way. The lift and iROAS section of the framework has the full math.
Example (hypothetical, simulated numbers): I pause non-brand search for six weeks in holdout regions that make up about 20% of sales. I'll read it over eight weeks: the six-week pause plus a two-week observation period. The treatment regions say the holdout regions should sell about $500,000 in that window, and my MDE range for this design tops out around 4%, so I size to 4%. To keep non-brand, I need it to beat breakeven, 1.82x. So I need to switch off at least 4% × $500,000 ÷ 1.82, or about $11,000. I'm pausing $15,000, so the test can see it. That size lets me tell a breakeven-sized lift from zero. Telling it apart from breakeven itself would take a bigger test.
At the read date, the treatment regions say the holdout regions would have sold $500,000 with search on. They sold $470,000. That's $30,000 in incremental revenue on $15,000 of paused spend, a 2.0x iROAS. The range (a 90% interval) runs from 1.0x to 3.0x. Over the same weeks, Google Ads reported 4.0x for non-brand in the treatment regions.
Numbers in this example are simulated for illustrative purposes. They are not a client result or a benchmark.
Keep, trim or cut: what happens next?
So which one? Whichever one the rule you wrote down before the test says.
Here's one way I read it. Before launch, write down the iROAS you expect. If the result lands within about ±20% of that number, act on it. That band can widen or narrow with your appetite for change and your volume.
Far above or far below? Make a digestible change and retest, or design a different test. Don't go all in on one read.
In the example above (still simulated), say I'd written down 3.0x as my expected iROAS. My band runs from 2.4x to 3.6x. The 2.0x result lands well below it, even though it still clears breakeven. So I trim a slice of non-brand spend and retest, instead of cutting it.
What mistakes ruin a PPC holdout?
Picture this: the test is set up perfectly. Then in week two, someone raises bids in the treatment regions, and someone else launches a promo in the holdout. Now the test is reading three changes at once.
- Pausing nationally. No comparison, no answer.
- Changing bids or budgets mid-test. Now you're testing two things.
- Calling it after a week. See the garden.
- Reading only platform conversions. They can't tell you what would have sold without the ads.
- Forgetting brand search and organic overlap. Some people who clicked your brand ad may have clicked your organic listing instead.
A holdout only works if everyone leaves it alone until the read date.
FAQs
Is it safe to turn off Google Ads to test it?
It's safer in a few regions than everywhere. A geo holdout limits the downside to the holdout regions and gives you a comparison. Agree on a safety rule before launch. If the holdout clearly hurts the business, turn search back on and stop the test.
Should I test brand search or non-brand search first?
Test whichever one the decision is about. Brand often catches people already on their way to buy. Non-brand is more likely to bring in new customers. Testing one at a time keeps the result readable. If you don't have a decision for either yet, that's the first thing to settle.
How many regions do I need for a paid-search holdout?
There's no fixed count. You need enough holdout regions for the lost sales to stand out from normal noise, and treatment regions that have tracked them closely in the past. Your own sales history decides the rest. The feasibility checks show how to test it.
How long should a paid-search holdout run?
Calculate MDE and power over a proposed length, then make sure the test covers several of your customers' buying cycles. My rule of thumb is 4 weeks minimum. Non-brand search usually needs longer than brand, because its path to purchase is longer. Add a short observation period after search comes back on.
Does Performance Max respect location exclusions?
Google's help page on advanced location options lists Performance Max and Shopping among the campaign types where location exclusions use "Presence" (people in, or regularly in, the excluded area). Google also says location targeting is a best effort and not 100% accurate. Add the exclusions to every campaign type, and check Google's current help page the day you launch.
What if the result is inconclusive?
Then it was either underpowered or, if you can show it was well powered, effectively negative. Either way it's not a clear win. The next step is a test you can power with confidence. That may be the same question with bigger regions or a longer window, or a different question.
Big picture
Write the decision. Pick the regions. Pause one thing, cleanly, for long enough. Read it as a range against your breakeven, and act on the rule you agreed to.
That's a lot more work than flipping a switch. But it turns "do we even need Google Ads?" into a question your own sales can answer.
Test the pause in a few places before you make it everywhere.
Need help? Speak with me here.