Incrementality Testing: How to Know If Your Ads Are Actually Working

Every performance marketer has faced this: retargeting ROAS looks excellent at 8x, but when you pause the campaign, sales barely drop. The reason is that most retargeting converts people who were going to buy regardless. It shows up last in the attribution path and claims credit. Last-click and even data-driven attribution models systematically overvalue bottom-funnel tactics that intercept existing intent. Incrementality testing measures the actual lift your ads produce: what would have happened without the campaign vs what happened with it.

The two main approaches to incrementality testing

Geo holdout tests split your market into test and control regions. You run ads normally in test regions and withhold them (or reduce spend significantly) in control regions for 2–4 weeks, then compare conversion rates between the two groups adjusted for baseline differences. This is the most statistically reliable method but requires meaningful geographic reach and a business that isn't too concentrated in a single city. Platform-native holdout tests (Meta's Conversion Lift, Google's Conversion Lift) use a randomly selected user holdout: a percentage of your campaign's target audience sees no ads and serves as the control group. These are easier to run but you have to trust the platform's methodology, which creates a conflict of interest since the platform benefits from showing high lift.

How to run a simple geo holdout test

• Choose 6–10 comparable cities or regions. Match on population, income level, and historical conversion rate
• Split them randomly into test (ads on) and control (ads off or significantly reduced) groups
• Run for at least 2 weeks to account for weekly seasonality; 4 weeks is better
• Compare conversion rates per capita between test and control, adjusted for any pre-test differences
• Calculate incremental ROAS: incremental revenue / ad spend in the test regions
• If incremental ROAS is substantially lower than reported ROAS, your attribution model is overstating the impact of the campaign

What to do with incrementality data

Incrementality tests rarely show that campaigns have zero effect, but they frequently show that effect is 40–60% lower than attributed ROAS suggests. Use this as a calibration factor for your reported metrics, not as a reason to cut campaigns entirely. If retargeting shows 2x incremental ROAS against a reported 8x, you now have a more accurate picture of what the channel is worth. Reallocate budget from over-attributed channels toward upper-funnel activity that generated the demand retargeting is now closing. That is where the real impact is.