What Questions Should I Ask an Attribution Vendor, and How Do They Validate Model Accuracy?

The best time to ask hard questions about an attribution tool is before the demo ends and the contract begins. A polished interface is easy to show. Reliable capture, reconciliation, and useful investigation take more work.

These questions will not name a universal winner. They will tell you whether a product fits the measurement job in front of your team.

What should you ask an attribution vendor before you buy?

Ask questions that show whether the tool fits the decision in front of you. It depends on that decision, so start there. Then get specific. How does the vendor check its model against geo tests? What share of your orders does it capture, and can you match them by order ID? Can you open a credited order and see the rule behind the credit? The ten questions below cover each one, with a good sign and a warning sign.

1. Which of our decisions will this improve?

Name the decision before you name vendors: whether to scale a channel, understand a long consideration cycle, separate new from returning demand, or investigate a claimed conversion spike. A tool that cannot improve a live decision is extra reporting.

Good sign: the vendor asks which decision you are trying to make, then shows the view you would use to make it.
Warning sign: the answer is a feature tour.

2. How do you validate your model's accuracy?

Every attribution model tries to communicate some level of causality. A credit to paid social is a claim about what paid social caused, whether or not the vendor says so. The only way to validate that claim is to pair the model with a geo test that demonstrates causality: change exposure in some markets, compare them with markets that stayed the same, and see whether the model's read holds up.

So ask to see it. Which experiments has the model been checked against, and what happened when they disagreed?

Good sign: the vendor calibrates or validates its model against geo experiments, and can explain how.
Warning sign: “validation” with no experiment behind it.
Red flag: a vendor that uses its deterministic pixel-based attribution and claims whatever it reports is incremental.

If you have never run a geo test, the incrementality testing framework shows how to plan one around a budget decision.

3. What share of our orders do you actually capture, and can we reconcile them?

Don't expect complete coverage. In Eric's experience, it is normal for third-party client-side tracking to capture about three-quarters of events, purchases included. If you also sell on Amazon or by phone, the tool sees an even smaller share of the total business. That figure is a description of what is ordinary, not a pass mark.

What matters is knowing which orders drop off. Ask the vendor for the order IDs it processed and match them against the order IDs of record in your own order management system. Then look past the click: can you inspect product views, checkout starts, purchases, refunds, and payment-provider handoffs? A tracked click with a blank order needs a different investigation than a missing click. First-party cookies and first-party code can close part of the gap.

Your business already has a record of orders and revenue. Do not skip the discrepancy because the multi-touch chart looks more interesting. For where those records usually split, read why GA4 and Shopify revenue disagree.

Good sign: order-level data you can match to your own order IDs, with coverage stated against a defined total.
Warning sign: a coverage figure with no denominator, or no way to see which orders are missing.

4. What happens when tracking drops?

Ask how the product handles missing UTMs, click IDs, consent changes, and browser controls. You need to know which records are observed, inferred, or absent, not merely whether the dashboard can fill a chart.

Good sign: each record is labeled observed, inferred, or missing.
Warning sign: gaps are filled silently and the chart looks complete.

5. How do you handle two platforms claiming the same order?

As of October 2026, Meta and Google can each report the same purchase as their own conversion. Add up platform-reported conversions and the total can exceed the orders you actually took. Ask how the tool decides which touchpoint gets the credit for one order ID, and whether total channel credit adds back to your order count.

Good sign: each order is credited once, the rule is visible, and channel totals reconcile to orders of record.
Warning sign: channel totals that add up to more orders than the business took.

6. Can we see the model and the journey detail behind a credit?

A channel can drive a high share of new visitors who bounce, or a smaller number who return and buy later. Pages viewed, checkout progression, and visit timing give needed context to an attribution label. Ask, too, whether you can see the rules or weights that produced a credit, and whether you can adjust them.

Good sign: you can open a credited order, see the journey behind it, and see the rule that assigned the credit.
Warning sign: the vendor can't explain how credit gets assigned, and you can't see the journey.

7. Does it integrate with the systems we actually use?

Confirm the order platform, analytics, ad platforms, warehouse, and any unusual source that matters to your business. “Integration” can mean a surface-level connector or a usable record you can cross-reference later.

Good sign: the integration produces records you can join to your own data by order ID.
Warning sign: “we integrate with that” means only a logo on a slide or a summary feed.

8. Can we investigate an unusual claim?

Suppose a podcast partner claims a run of orders. Can the team compare order IDs across systems, inspect later conversion channels, and look at first-visit timing? A rigid dashboard may report a number without helping you understand it. This is where order-ID-level data from question 3 pays off.

Good sign: you can trace a claimed order from first visit to purchase in your own records.
Warning sign: the only answer to “why?” is the dashboard number again.

9. What can't the tool observe?

If your portfolio includes TV, radio, podcasts, retail, or brand work, ask the vendor to be explicit about what the tool cannot observe. Orders that happen off your site, such as on Amazon or by phone, also sit outside site tracking. A click-based model should not quietly become the judge of all spend.

Good sign: the vendor names its blind spots before you ask.
Warning sign: every channel gets a confident number, including the ones nobody clicks.

10. How does identity hold together across sessions and devices?

Ask what first-party data, cookies, login signals, and event records are used to recognize a return visitor. Then ask what happens when the shopper changes devices or the identifier disappears. The limitation is part of the product.

Good sign: a plain explanation of which signals link visits and what happens when they break.
Warning sign: “we see every customer across every device.”

How to score the answers

What should you ask a vendor about how they validate their attribution model's accuracy?

Ask how they validate their attribution model's accuracy against an experiment on your own data. Accuracy means the tool's credit for a channel holds up against the lift a geo test or holdout measures. Then check that its orders reconcile to your order system. Until an experiment backs it, treat the credit as a hypothesis; no model owns the truth.

Here are three checks to run.

1. Do the orders match? Ask for the order IDs it processed. Match them to your order management system. You want a denominator, so you can see the orders it missed.
2. Does the credit match a test? Compare the tool's credit for a channel with the lift a geo test or holdout measures for that channel. When a test covers the question, the test anchors it.
3. Is pixel credit being sold as incremental? Deterministic pixel-based attribution shows which touches came before a sale. It doesn't show what the media added. If a vendor calls it incremental, that's a red flag.

Ask about views, too. Last click undervalues upper-funnel channels, so ask how the tool credits views and media people never click.

If the same test will also feed an MMM, read what an incrementality test can calibrate in an MMM.

What the answer may be

Sometimes a tool is a good fit. Sometimes the next investment is first-party event capture, identity work, or a cleaner warehouse layer. Sometimes the decision is about causal impact and needs a test. The goal is to buy the next useful capability, not a dashboard with a familiar logo.

If you do buy, plan the follow-through before launch. Choose a reconciliation check against your order IDs, a few known customer journeys, and a decision you will review once the tool has enough history. Where the stakes are high, pair the tool with a geo test and run an incrementality test rather than treating the new model as final proof. A dashboard can be accurate about what it saw and still point the budget the wrong way; six reasons that happens are worth reading before the first reallocation.

For the longer decision framework, read how to choose an ecommerce attribution tool without buying another dashboard.

Attribution vendor evaluation: common questions

What questions should I ask an attribution vendor during evaluation?

Start with how the vendor validates accuracy. The answer you want is a model checked against geo experiments. Then ask for the order IDs it processed so you can reconcile them against your own order system, and ask how it deduplicates orders, handles tracking loss, shows the journey and rules behind a credit, investigates an unusual claim, and keeps identity together. Ask what it cannot observe, and tie every answer to the decision the tool has to support.

How do I check whether an attribution vendor's model is accurate?

Pair it with a geo test that demonstrates causality. Every attribution model implies some causal claim, and an experiment is the only way to check it. A vendor that offers validation without an experiment, or claims its deterministic pixel-based attribution is incremental, is a red flag. The incrementality testing framework shows how to plan the test. See how to check accuracy step by step.

How much of my data should an attribution tool capture?

In Eric's experience, about 75% of events, purchases included, is normal for third-party client-side tracking. The share of total business is lower if you also take orders on Amazon or by phone. Reconcile the vendor's order IDs against your order management system to see what drops off. First-party cookies and first-party code can close part of the gap.

Can an attribution tool measure incrementality?

Not on its own. Every attribution model implies some causality, but only an experiment, such as a geo test, demonstrates it. See what MTA can and cannot answer and the incrementality testing framework.

Should we replace GA4 with an attribution tool?

Usually that is the wrong first question; start with the decision the tool has to support. The ecommerce attribution tool evaluation guide answers this one in its FAQ.

Want to see which tools fit each method? Here’s a table of tools by method.