Find out which ads bring in sales you wouldn't get anyway
Ad platforms and email tools count every sale they touched. A test shows how many of those sales they caused. We design the test, run it on your Shopify numbers and turn the answer into a budget decision.
Free, 30 minutes, and you'll leave with at least one clear next step.
What is incrementality testing?
Incrementality testing measures how many extra sales an ad, channel or email caused, by comparing a group that saw it with a similar group that didn't. The difference between the two groups is the lift. Everything else is sales you would have made anyway.
It answers a question tracking can't. Our ad tracking and reporting work tells you which sales followed which ads, as accurately as the data allows. A test tells you which of those sales the ad actually caused. Say a shopper searches your brand name, clicks a Shopping ad and buys. That counts as an ad sale, but she was already on her way to your store. Retargeting ads and cart emails often reach people who were going to buy. The platforms aren't lying. They just count anything they touched.
What the dashboards report and what a test shows
Platform and Klaviyo reports
- Sales that happened after someone clicked, viewed or opened
- Each platform counts with its own rules, so totals overlap
- Repeat buyers and brand searchers counted the same as new customers
- Orders counted before returns come out
An incrementality test
- Sales that only happened because of the ad or email
- Measured in total Shopify orders, which can't double count
- A clear read on whether a channel finds new customers
- Judged on revenue after returns and refunds
The tests we run
- Geo holdout. Turn a channel off or down in a set of matched states or metro areas and compare total Shopify sales with the areas where it kept running. Works for Meta, Google, TikTok and YouTube.
- Brand search pause. Stop bidding on your own brand name for a few weeks and see how many of those orders still arrive through the free listings.
- Retargeting holdout. Hold a share of site visitors out of retargeting ads and compare how many buy.
- Email and SMS holdout. Keep a small random group out of a flow, like welcome or cart reminders, and compare orders. See our Klaviyo agency and SMS marketing pages.
- Platform lift studies. Meta's Conversion Lift and Google's own lift experiments split audiences inside the platform. We set them up when you meet the platform's current spend minimum, then read the results against Shopify.
- Spend step test. Raise or lower a channel's budget sharply for a set period and watch what total sales do. It's quick and rough, so we use it to decide what to test properly next.
How a test runs from question to budget change
- Week 1
Pick one question
One question per test. For example, we might ask whether retargeting is worth what you spend on it, or whether Performance Max finds new customers or just your regulars.
- Week 1
Design it
We choose the test type, the test and control groups, how long it runs and the result that would change your budget, before anything starts.
- Week 2
Check the baseline
We confirm the two groups have sold alike in recent weeks and flag anything that would muddy the read, like a planned sale or a stockout.
- Weeks 2 to 8
Run it
Most tests run two to six weeks, depending on how many orders you get a day. We keep everything else steady while it runs.
- Final week
Read it and act
We compare sales after returns, show the lift and how sure we are, and recommend a budget change. After that, we pick the next question.
Mistakes that ruin a test
A bad test is worse than no test, because it gives a confident wrong answer. These are the ones we plan around.
Running it during a sale
A Black Friday or launch week swamps any difference between groups. We schedule tests for normal weeks.
Stopping when the first week looks good
Early results swing a lot. The end date is set before the test starts, and we stick to it.
Reading lift from the platform's own numbers
If you pause Meta in some regions, Meta's report can't tell you what happened there. We read the result in total Shopify orders by region.
Ignoring returns
A channel that drives orders people send back can look like a winner. We take returns out first, using the method in our returns guide.
Results from brands we run ads for
First Stop Fashion more than doubled its online sales in a year
Read itVeiled Collection went from pop-up shops to more than 50,000 new customers online
Read itBaked Bouquet grew from pop-up sales to more than 10,000 new customers
Read itDoes this replace our normal reporting?
No. Tests and tracking work together. Day-to-day reporting tells you how campaigns compare inside each platform, and it's fast. A test, run a few times a year on your biggest channels, tells you how much to trust those reports. We use the test result to adjust how we read the dashboards. If a test shows retargeting adds only a small share of what Meta credits it with, we weigh its reported numbers down to match.
We already judge accounts on store totals day to day. First Stop Fashion's Meta and Google budgets follow total store sales, not each platform's claims. Veiled Collection's ads use synced customer lists, so spend goes to people who haven't bought yet. A test takes that one step further. More of our ad work is on the paid ads page.
Is our brand big enough to test?
If you get enough orders a day that a normal week doesn't swing wildly, you can run a useful test. Smaller brands start with simpler tests, like a brand search pause or an email holdout, which need fewer orders to read. Geo tests and platform lift studies need more volume and more spend.
Testing suits most online brands that spend across two or more ad channels, from fashion to beauty to food and gifting. It pairs well with email marketing, because flows are often the easiest place to run a first holdout. On the call, we'll tell you which test fits your order volume.
Incrementality testing questions
How is incrementality different from attribution?
Attribution assigns credit for sales that already happened to the ads people touched. Incrementality testing compares a group that saw an ad with one that didn't, to find the sales the ad caused. We use both. Attribution runs every day, and tests check how far to trust it.
Will a holdout test cost us sales?
It can cost a few, for a short time, in the group that doesn't see the ad or email. If the channel turns out to add little, the test saves far more than it costs. We keep the held-back group as small as the test allows.
How long does an incrementality test take?
Most run two to six weeks, plus a week to design and a week to read. Brands with fewer daily orders need the longer end to get a clear answer.
Do we need special software to run these tests?
Usually not. Most tests run on your Shopify data, the ad platforms' own settings and a careful spreadsheet or dashboard. Meta and Google don't charge extra for their lift studies, but each has its own spend minimums and rules. If you spend across many channels at large budgets, a paid measurement tool can be worth it, and we'll say so.
Which channel should we test first?
The one with the most spend and the least certainty. For many Shopify brands that's retargeting, brand search inside Performance Max, or the welcome and cart email flows.
What if the test shows a channel isn't working?
We don't shut it off on one result. We cut its budget to the level the test supports, move the rest to channels that showed lift, and retest later. Results change as creative, offers and seasons change.
Related
More services
Pick the channel you trust least
On the call we'll look at where your reported results and your Shopify sales split, and sketch the first test that would settle it.