An incrementality result can change how much weight you give an MMM's channel reading. It does not automatically prove the model correct, turn one test into a year-round allocation rule, or require an overnight budget shift. The useful question is narrower: does this test say enough about the channel under the conditions your model is meant to guide that it should inform the model's prior or interpretation?
What “calibrate” should mean here
It can change how much weight you give the model's reading for the channel you tested. In a Bayesian MMM, a well-run test can inform that channel's prior. Whether it should depends on whether the test earned that role. Was the treatment-control baseline stable before launch? Did the test have enough power? Do its conditions match the period the model will guide? Even then, one test doesn't prove the model right. It doesn't set a year-round rule or call for an overnight budget shift either.
In a Bayesian MMM, a prior is an explicit starting belief about a channel's performance. An incrementality experiment can be one source for that belief. Google's research on Bayesian MMM calibration describes using experimental results or domain knowledge to inform an ROI prior, rather than treating a test result as a mechanical replacement for the model's estimate.
A test can provide evidence for a channel-specific prior, compare two operating strategies, reveal that the model is missing a condition, or produce a result too weak or mismatched to generalize. Calling each outcome “calibration” hides the decision the team still has to make.
1. Start with whether the test earned authority
A result deserves more weight when the team can explain the treatment and control, the business outcome, the counterfactual, the result interval, and why the design could detect a meaningful change. A high-confidence result from a weak comparison is not enough.
Before using a geo result to inform a model, review whether the treatment and control relationship was stable before the test. That means inspecting the control-to-treatment scaling relationship across the preceding 30 days, 90 days, six months, and year. These windows are a practical review process, not universal statistical cutoffs. Their purpose is to expose an unexplained shift before one pre-period becomes the claimed fair baseline.
Power and confidence matter as well. A test needs a meaningful chance of detecting a non-null effect before a null or weak result should reshape a channel reading. Geo inference can be difficult when units are few or heterogeneous, and budget constraints can create interference between them. Chen and Au's paired-geo research addresses those risks in one framework; it is not a universal method prescription.
2. Check whether the test speaks for the modelled period
An experiment and an MMM may use the same word—ROI—while describing different conditions. Meridian's calibration guidance makes the relevance questions explicit: time period, population, media granularity, campaign complexity, counterfactual, and spend scale can all differ between a test and the model.
A Black Friday geo test shows why this matters. If a channel behaved unusually during a promotion and that behavior is not expected across the year, the result may be real for Black Friday without being a sound year-round channel prior. The same concern applies to a strategy test, such as comparing one-day-click and seven-day-click approaches. The result can guide which strategy to adopt; it should not automatically become a statement about the channel's standing ROI.
Upper-funnel media deserves extra caution. A short test may be too short to observe the effect the business actually cares about. A low result from four weeks of media can describe that four-week intervention without establishing that the channel has low longer-term value. The next move may be a longer observation, a better-matched question, or keeping the result as a bounded signal.
Calibration Claim Ledger
Use this record before saying a test “anchors,” “constrains,” or “validates” an MMM.
| Record | What to write down | Why it changes the decision |
|---|---|---|
| Business decision | The channel, budget move, and consequence of being wrong. | Keeps the analysis tied to an action rather than a model score. |
| Test result | Treatment/control contrast, outcome, counterfactual, interval, power, and confidence. | Prevents a point estimate from standing in for the full result. |
| Baseline | Pre-test scaling across 30, 90, 180, and 365 days, plus any explanation for changes. | An unexplained shift can make the comparison less defensible. |
| Candidate model input | The specific ROI prior, scenario assumption, or interpretation the test may inform. | One channel result does not calibrate the whole model. |
| Relevance gaps | Population, geography, season, media scope, spend scale, duration/carryover, and estimand differences. | These determine whether to widen uncertainty, keep the result separate, or not generalize it. |
| Next action | A staged change, model review, follow-up test, or decision to wait. | Turns the work into a controlled business choice. |
Meridian documents automated adjustments for spend, recency, and duration when translating experiments into priors. Those adjustments are useful examples of the translation problem, not proof that automation makes every test representative. Its past-experiment guidance shows why a result's scale, age, and carryover assumptions should remain visible.
3. When the test and model disagree
A credible test can be stronger evidence than a conflicting MMM reading. MMMs work from historical observational data and can struggle when a channel lacks independent variation, the data do not capture the relevant timing, or the model cannot separate a channel from correlated demand. Chan and Perry outline the data and causal-inference constraints that make this work difficult.
That does not mean every test settles the question. Review the design and relevance first. If the test holds up, use the disagreement to ask what the model is missing: insufficient variation, an omitted condition, a timing problem, a carryover assumption, or a different estimand. The productive outcome is not forced numerical agreement. It is a clearer statement of what the business knows and which next test or model change could reduce consequential uncertainty.
4. Take a bounded next step after a strong result
Suppose a credible test suggests a channel has more headroom than the current model or budget assumes. A model optimizer may imply a large increase. The business does not have to make that full move at once.
It can raise spend in steps within the range the evidence can reasonably inform, inspect the whole-business response, and decide whether to continue. It can also run a follow-up geo test in a limited footprint at the new spend level. This does not predict that the higher spend will work. It buys additional evidence while keeping the size of a mistake proportionate to the business's risk appetite.
Choose the correct next move
- Credible and relevant result: review how it changes the channel prior or interpretation, retain uncertainty, and inspect model sensitivity before a budget move.
- Useful but contextual result: use it for its original strategy or period, document why it does or does not represent the broader channel, and consider a staged change or follow-up test.
- Underpowered result, unresolved baseline, or major mismatch: do not generalize it into the MMM. Repair the evidence, run a better-matched test, or wait.
If a test result and model reading point to different budget actions, the first conversation should not be “Which tool wins?” It should be “Which conditions does each result actually describe, and what decision can we safely make from that?”
Sources and limits
This guide synthesizes research, method documentation, and Eric's practical review process. It is not a universal calibration procedure, a guarantee that a test validates an MMM, or a forecast of any channel's ROI.
- Media Mix Model Calibration With Bayesian Priors, Zhang et al. (2024).
- Meridian: Calibrate treatment priors, reviewed September 27, 2026.
- Meridian: Set custom ROI priors using past experiments, reviewed September 27, 2026.
- Challenges and Opportunities in Media Mix Modeling, Chan and Perry (2017).
- Robust Causal Inference for Incremental Return on Ad Spend with Randomized Paired Geo Experiments, Chen and Au (2022).