On this page6 sections
A WhatsApp campaign incrementality test sends a campaign to most of an eligible audience and deliberately withholds it from a random, stable slice. The gap in orders between the two groups is what the campaign caused; everything else would have happened anyway. Getting that number right is a design problem, settled before the first message goes out.
This playbook covers the design choices: which question you are answering, how to assign and size the holdout, which outcome and window to fix, and how to turn the result into a cost per incremental order. For the definition and the basic lift formula, see the holdout group glossary entry. If you need attributed ROI for a monthly report, measuring WhatsApp campaign ROI covers it; a holdout tells you how much of that ROI the campaign actually caused.
Decide which question the test answers
Most disappointing holdout tests fail at this step. "Did the Diwali broadcast work?" and "Is WhatsApp marketing worth its budget?" need different holdouts, and a single 10% slice cannot answer both.
| Question | Who is held out | How long the assignment lasts | What you learn |
|---|---|---|---|
| Did this campaign add orders? | A random slice of this campaign's audience | One campaign plus its measurement window | Lift and cost per incremental order for one send |
| Does WhatsApp marketing as a whole pay? | A random slice of all marketing-eligible contacts, excluded from every marketing send | A full quarter or longer | Programme lift, including fatigue from repeated sends |
| Does the second touch add anything? | Everyone gets touch one; a random slice of non-buyers skips touch two | One sequence | Whether follow-ups earn their cost |
The campaign holdout is the easiest to run and the usual starting point. The programme holdout is the one finance teams ask for at budget time, because a string of campaign-level lifts can each look positive while the programme as a whole mostly moves orders from one week to another. If you can run only one, run the programme holdout for a quarter.
An A/B test answers a different question, which version works better, and it can sit inside the treated group of any of these designs. The abandoned cart playbook shows the same holdout idea applied inside a single automated flow.
Assign once, keep it stable
Assignment rules decide whether the result survives scrutiny from your finance team.
- Filter first, then split. Apply consent, opt-outs and your usual suppression rules to build the eligible audience, then assign. Splitting first and filtering later removes different kinds of people from each side.
- Use a deterministic hash. Assign by hashing the contact ID with a fixed test name, for example
hash(contact_id + "q4-programme") mod 100 < 10for a 10% holdout. The same person lands in the same group every time the audience is rebuilt, and a new test name gives a fresh, independent split. - Record the group on the contact at assignment. A tag such as
holdout-q4makes suppression mechanical and lets anyone audit the split later. - Never re-draw mid-test. Re-randomising for each campaign leaks programme-holdout members into treatment and quietly ends the test.
How big the holdout needs to be
Size the holdout from the smallest lift worth acting on, not from habit. A standard two-proportion calculation at 95% confidence and 80% power gives a working figure. Suppose 4% of your eligible customers order within 14 days without any message, and you want to detect a rise to 5%:
| Split (treated : holdout) | Holdout contacts | Treated contacts | Total eligible |
|---|---|---|---|
| 50 : 50 | about 6,750 | about 6,750 | about 13,500 |
| 90 : 10 | about 3,430 | about 30,850 | about 34,300 |
These are arithmetic on assumed rates, not benchmarks, so plug in your own baseline. A lower baseline or a smaller lift you care about pushes the numbers up quickly. If your list is smaller than the table needs, pool several campaigns into one programme holdout rather than reporting a lift your sample cannot support.
Fix the outcome and the window before you send
Write down three things before anyone is assigned, and do not change them after you see the data.
- The outcome. Orders, renewals or bookings from every channel (website, app, store counter, cash on delivery) matched to the contact by phone number or customer ID. Clicks and replies cannot be the outcome, because the holdout never receives a message to click.
- The window. Match it to how long customers take to act: a weekend flash sale might need 7 days, a considered purchase 14 to 30. Count from the scheduled send time for both groups.
- The read-out date. Checking daily and stopping on the first good-looking day inflates false positives. Read out once, on the date you wrote down.
Keep failed sends in the treated group
Compare everyone assigned to each group, not only the people whose message was delivered or read. On WhatsApp this matters more than on most channels, because some treated sends fail by design:
- WhatsApp may limit how many marketing templates a person receives from any business. A withheld message fails with error code 131049, per Meta's per-user marketing limits documentation.
- Meta runs a marketing message experiment in which a very small percentage of WhatsApp users get no marketing templates from any business unless a customer service window or free entry point window is open. Those sends fail with error code 130472, and resending returns the same error.
Dropping these contacts from the treated side removes people who were never reachable and flatters the result. Keep them in. The lift you report is then the lift of deciding to send, which is the decision you are actually making. The per-user marketing limit entry explains how that cap behaves across campaigns.
Turn lift into cost per incremental order
Meta charges only for delivered messages, per its pricing documentation, so the cost side uses delivered counts while the lift side uses assigned counts. Put the two together like this:
- Incremental orders = (treated order rate − holdout order rate) × treated contacts.
- Campaign cost = delivered marketing messages × ₹0.94 (InfiQ's India rate per delivered message, ex-GST) + discount value on every redeemed order in the treated group + any agency or creative cost.
- Cost per incremental order = campaign cost ÷ incremental orders.
The discount line is the one teams leave out. A coupon is redeemed by customers who would have ordered anyway, and the holdout is how you see that.
Worked example: a hypothetical Jaipur home-décor brand
The brand and every figure below are illustrative. It has 40,000 marketing-eligible contacts and holds out 10% from a festive campaign carrying a 10%-off code.
- Treated: 36,000 contacts, of whom 34,560 had the message delivered. Holdout: 4,000 contacts, no message.
- Orders within 14 days: 1,800 treated (5.0%), 160 holdout (4.0%).
- Incremental orders: 1.0 percentage point × 36,000 = 360. A last-click report would have credited the campaign with 1,800.
- Redemptions: the code was used on 1,200 orders at an average order value of ₹1,500, so ₹1,80,000 in discounts.
- Cost per incremental order: (34,560 × ₹0.94 + ₹1,80,000) ÷ 360.
The discount alone works out to ₹500 per incremental order before a single message is paid for. Run the next campaign without a code, with its own holdout, and you learn whether the discount or simply the reminder drove the lift.
Report the uncertainty with the number. With these group sizes, the 95% confidence interval on the 1.0-point lift runs from roughly 0.35 to 1.65 points, or somewhere between about 125 and 595 incremental orders. A decision that only makes sense at the top of that range is not yet a decision. The campaign ROI calculator handles the cost side once you have the incremental count.
Pitfalls that are specific to WhatsApp
| Pitfall | What it does to the result | Fix |
|---|---|---|
| Another team broadcasts the same offer to the holdout | The holdout behaves like the treated group and lift shrinks | Exclude the holdout tag from every marketing send in the test period |
| The offer reaches the holdout by SMS or email | Same contamination through another channel | Hold the offer back on all channels, or test the channels together |
| Agents paste the offer when holdout customers message in | Part of the holdout gets treated | Answer their questions, but do not send the campaign offer |
| Utility messages are paused for the holdout | You measure the loss of order updates, not the campaign | Keep utility and service messages flowing to everyone |
| Only delivered or read contacts are compared | Lift is inflated | Compare everyone assigned |
| The holdout is re-drawn for every campaign | Programme lift becomes unmeasurable | Hash on a fixed test name and keep the tag |
One advantage is easy to miss: a holdout protects you against seasonality, because both groups live through the same festive week. A before-and-after comparison does not, which is why a Diwali campaign compared with the first fortnight of October tends to look better than it was.
Running holdout tests on InfiQ
InfiQ's broadcast campaigns build audiences from tags, attributes and purchase history, so a holdout tag can be excluded from every send in the test period. Campaign analytics report delivered, read, replied, clicked and converted for each campaign, which gives you the delivered count for the cost side. The order comparison between groups runs on your own order data, joined by phone number or customer ID.

