Skip to content

Glossary drawing · Growth & marketing

Holdout Group

Also called: Holdout · Hold-out group · Holdout test · Incrementality test · Campaign control group

Note 1 —A share of an audience deliberately left out of a campaign so the difference in their behaviour shows the campaign's true incremental lift.

Term
Holdout Group
Abbreviation
—
Category
Growth & marketing
Level
Expert
Reviewed
Sheet no.
GLO-GRO-592
Drawn by
InfiQ Glossary

Drawing 1 · Interactive

Holdout versus messaged: is the lift real?

14 / 14

Metric · run 1

Renewed within 14 days

Visitors per variant12,000 / 12,000

  • AHoldout (not messaged)28.0%± 0.8 pts, 95% interval 27.2% to 28.8%
  • BMessagedWinner30.5%± 0.8 pts, 95% interval 29.7% to 31.3%
  • Observed rate
  • 95% interval (± 1.96 × standard error)
  • True rate (unknown in a real test)
  • Where the leaders overlap

B (Messaged) wins

Observed 30.5% (+9.1% vs A). Its 95% interval no longer overlaps the other one, so the gap is very unlikely to be chance.

  • AHoldout (not messaged)Control

    Customers chosen at random and left out of the campaign. Their rate is what happens without the message.

    3,355 of 12,000 converted

  • BMessaged

    Customers sent the renewal-offer template, counted whether or not it was delivered or read.

    3,659 of 12,000 converted · +9.1% vs A

Simulated renewals for customers left out of a campaign and customers who were messaged. Watch the confidence intervals narrow as the sample grows, then run it again with a fresh random draw.

12,000 visitors per variant. B (Messaged) wins. A: 28.0%, 95% interval 27.2% to 28.8%; B: 30.5%, 95% interval 29.7% to 31.3%.

What is a holdout group?

A holdout group is a randomly chosen share of a campaign's audience that is deliberately not sent the campaign. Everyone else — the treated group — gets the message as planned. After the campaign window you compare what the two groups did: purchases, renewals, bookings, repayments. Because the split was random, the holdout shows what the treated group would have done without the message, and the difference between them is the campaign's incremental lift.

This matters because attribution based on clicks, replies or a last-touch window credits the campaign with every conversion that follows a message, including customers who were going to buy anyway. A holdout separates the two. If 5.5% of treated customers bought within seven days and 4.2% of the holdout bought too, the campaign caused about 1.3 percentage points of conversion — a relative lift of roughly 31% — not 5.5%.

A holdout is not the same as A/B Testing. An A/B test sends two or more versions and asks which works better; a holdout asks whether sending works at all, and how much it adds over silence. The two combine well: hold out 10% and split the rest between versions A and B. Platforms run holdouts too — Meta's marketing message experiment withholds marketing templates from a very small share of WhatsApp users, and sends to them fail with error 130472.

Key takeaways

  • A holdout group is a random slice of the audience that is deliberately not messaged, so a campaign's incremental effect can be measured.
  • Lift is the treated group's conversion rate minus the holdout's, which strips out the people who would have converted anyway.
  • Assign the holdout at random before sending, and compare everyone selected for the send, delivered or not.
  • An A/B test compares versions of a message; a holdout tests whether sending at all makes a difference.
  • Holdout members cost nothing to message, but each one may be a lost sale, so size the group deliberately.

Example: Does the renewal nudge earn its cost?

A Bengaluru broadband provider sends a renewal-offer template to 54,000 customers whose plans expire next month and holds out 6,000 chosen at random. Within 14 days, 31.0% of the treated group renew against 28.4% of the holdout. The campaign added about 2.6 points — roughly 1,400 renewals — rather than the 16,700 its dashboard had been crediting. The team keeps the campaign, because 1,400 renewals are worth far more than 54,000 deliveries cost, but drops the third reminder, whose own holdout test showed no measurable lift.

Operating sequence

How does a holdout group work?

  1. Fix the audience, outcome and window

    Decide who is eligible (for example, opted-in customers who bought in the last year), the outcome (a purchase within seven days) and the measurement window before anyone is assigned.

  2. Assign at random, then freeze

    Split eligible contacts randomly — commonly 5–20% to the holdout — using a hash of contact and campaign ID so a re-run gives the same split. Never hand-pick the holdout.

  3. Message only the treated group

    Suppress the holdout from this campaign and from any substitute message about the same offer on any channel, while still sending them normal utility messages.

  4. Measure everyone you assigned

    Compare all treated contacts — including those whose message failed or went unread — with all holdout contacts. Dropping undelivered treated contacts biases the result; this is the intention-to-treat rule.

  5. Compute lift and its uncertainty

    Absolute lift = treated rate − holdout rate; relative lift = absolute lift ÷ holdout rate; incremental conversions = absolute lift × treated contacts. Check the confidence interval before acting on a small difference.

Implementation

A holdout group in code

import hashlib
from math import sqrt
 
def in_holdout(wa_id: str, campaign_id: str, share: float = 0.10) -> bool:
    """Deterministic random split: the same contact and campaign always get the same answer."""
    digest = hashlib.sha256(f"{campaign_id}:{wa_id}".encode()).hexdigest()
    return int(digest[:8], 16) / 0xFFFFFFFF < share
 
def lift(treated_conv: int, treated_n: int, hold_conv: int, hold_n: int) -> dict:
    pt, ph = treated_conv / treated_n, hold_conv / hold_n
    diff = pt - ph
    se = sqrt(pt * (1 - pt) / treated_n + ph * (1 - ph) / hold_n)
    return {
        "treated_rate": round(pt, 4),
        "holdout_rate": round(ph, 4),
        "absolute_lift": round(diff, 4),
        "relative_lift": round(diff / ph, 3) if ph else None,
        "ci95": (round(diff - 1.96 * se, 4), round(diff + 1.96 * se, 4)),
        "incremental_conversions": round(diff * treated_n),
    }
 
# Count everyone assigned, delivered or not (intention-to-treat).
print(lift(treated_conv=16740, treated_n=54000, hold_conv=1704, hold_n=6000))
# -> absolute_lift 0.026, ci95 roughly (0.014, 0.038), about 1,404 incremental renewals

Hash-based assignment needs no stored random numbers, and the interval tells you whether the lift is real.

Specifications

A holdout group: key facts and limits

Typical holdout share
5–20% of the eligible audience
Assignment
Random, before the send, then frozen
Absolute lift
Treated rate − holdout rate
Relative lift
Absolute lift ÷ holdout rate
Incremental conversions
Absolute lift × treated contacts
Cost of messaging the holdout
None — nothing is sent
Meta's platform holdout
Marketing message experiment, error 130472

Design intent

Why does a holdout group matter?

On WhatsApp every marketing delivery is paid for — ₹0.94 per message in India — and every extra send carries some risk of blocks and opt-outs. Knowing whether a campaign creates sales, rather than merely being present when they happen, decides whether it should run at all, how often and to whom. A holdout turns attributed-revenue claims into incremental numbers a finance team can trust, and it is the only honest way to judge re-engagement and reminder campaigns aimed at people who might have come back on their own.

Failure modes

Common mistakes with a holdout group

  • Choosing the holdout by hand

    Holding out "customers who rarely buy" guarantees a flattering lift. Use random assignment from a hash or a seeded generator, and record it on each contact.

  • Comparing delivered contacts only

    Excluding treated contacts whose messages failed removes the hardest-to-reach people from one side only and inflates lift. Compare everyone assigned to each group.

  • Leaking the offer to the holdout

    If the holdout receives the same discount by email, SMS or a drip, the test measures nothing. Suppress the offer on every channel for the test window.

  • A holdout too small to read

    With 500 holdout contacts and a 3% base rate, a one-point lift disappears in noise. Size the holdout so the confidence interval is narrower than the effect you care about.

  • Measuring once and trusting it forever

    Lift decays as audiences tire of a recurring campaign. Keep a small rolling holdout on campaigns you run every week or month.

Recommended practice

Best practices for a holdout group

  • Store holdout membership on the contact so later analysis can reproduce the split.
  • Run a holdout and an A/B test together: hold out first, then split the rest between versions.
  • Use a window long enough to catch delayed conversions, such as 7 or 14 days.
  • Report incremental conversions per 1,000 messages to compare campaigns of different sizes.

Holdout Group FAQs

What is the difference between a holdout group and a control group?+
They are usually the same idea: a group that does not receive the treatment. Marketers say holdout when a slice of the audience is held back from a campaign to measure its lift; control is the general experimental term and is also used for the baseline version in an A/B test.
How big should a holdout group be?+
Commonly 5–20% of the eligible audience, sized so the result is statistically meaningful. Low base rates and small expected lifts need larger holdouts, while very large audiences can use a smaller share. Small audiences may need several campaigns pooled before the answer is reliable.
Is a holdout group the same as an A/B test?+
No. An A/B test compares versions of a message to see which performs better, while a holdout compares sending with not sending to see whether the campaign adds anything. Many teams run both at once: hold out a slice, then split the rest between versions.
Why do some WhatsApp marketing messages fail with error 130472?+
The recipient is in Meta's marketing message experiment, a platform-level holdout. Meta withholds marketing templates from a very small share of WhatsApp users unless a customer service window or free entry point window is open. These messages are not charged, and retrying gives the same error.
Does a holdout group cost money?+
It costs nothing to send, because holdout members receive no message, but it can cost sales. If the campaign truly works, the holdout converts less, so keep it only as large and as long as you need for a reliable answer.

See InfiQ pricing

* Per-message rates for India, ex-GST: marketing, utility and authentication effective 1 July 2026; service messages ₹0.145 since 1 October 2026 (they were ₹0 until 30 September 2026). Volume commitments earn discounts — final rate is confirmed on your account; applicable GST extra. Rates for other countries differ (see the international rate table on /pricing).

Put Holdout Group to work

InfiQ is an official Meta Business Partner. Start a 7-day free trial — no card needed — and see this running on your own number.

Start free trialBrowse all 550 terms