WhatsApp A/B Testing: Variants, Samples & Significance

01What is A/B testing?

A/B testing (also called split testing) is the practice of sending two versions of the same WhatsApp message — variant A, usually the current one, and variant B, the challenger — to two randomly chosen halves of an audience, then comparing one pre-agreed metric such as the click-through rate on a button, the reply rate or the conversion rate (orders per delivered message). The question is not only which version scored higher, but whether the difference is large enough, on enough recipients, that it would probably repeat.

On the WhatsApp Business Platform the variants are usually message templates. Some differences need no extra approval: header media (an image or video) and variable values are supplied at send time, so two images or two offers can run inside one approved template. Changing the fixed body text, the buttons or the footer means a second template, which goes through template approval (automatic, up to 24 hours) and should share the first one's category and language so the comparison stays fair.

A/B testing is easy to confuse with a holdout group. An A/B test compares two versions of a message against each other; a holdout deliberately sends nothing to a slice of the audience to measure whether messaging creates extra sales at all. Send-time optimisation is an A/B test of timing, and campaign analytics is where the results of every test end up being read.

Key takeaways

A/B testing sends two versions of one WhatsApp message to randomly split audiences and checks whether the better result is more than chance.
Change one thing per test and pick the metric before sending: button clicks, replies or orders per delivered message.
Detecting a 4% to 5% click-through lift needs roughly 6,700 delivered messages per variant at 95% confidence and 80% power.
Header media and variable values are set at send time, so testing two images needs no new template; new body text does.
A/B testing compares two versions; a holdout group measures whether sending at all makes a difference.

02 · How it works, step by step

How does A/B testing work?

Step 1
Write one hypothesis and one metric
For example: "A benefit-led first line gets more button taps than a price-led one." Choose the metric before sending — URL-button clicks per delivered message, replies, or orders within 7 days — and do not switch it afterwards.
Step 2
Size the sample
Smaller expected differences need bigger samples. Moving a 4% click-through rate to 5% needs roughly 6,700 delivered messages per variant for 95% confidence and 80% power; a 4% to 4.5% change needs about four times as many.
Step 3
Split the audience at random
Assign each contact to A or B by hashing their WhatsApp ID with the test name, so the split is random but repeatable. Never split by city, list source or sign-up date — that tests audiences, not messages.
Step 4
Send both variants in the same window
Send A and B at the same time, through the same phone number and the same template category, so time of day, festival traffic and number quality affect both equally.
Step 5
Wait, then test significance once
Let the decision window close (for example 48 hours for clicks, 7 days for orders), then run a two-proportion test on delivered counts. Ship B only if the gap is significant and large enough to matter.

03A/B testing in code

1import hashlib
2from math import sqrt
3from statistics import NormalDist
4 
5 
6def variant_for(wa_id: str, test_id: str) -> str:
7    """Random but repeatable: the same contact always lands in the same arm."""
8    digest = hashlib.sha256(f"{test_id}:{wa_id}".encode()).digest()
9    return "A" if digest[0] % 2 == 0 else "B"
10 
11 
12def two_proportion_test(hits_a, delivered_a, hits_b, delivered_b):
13    p_a, p_b = hits_a / delivered_a, hits_b / delivered_b
14    pooled = (hits_a + hits_b) / (delivered_a + delivered_b)
15    se = sqrt(pooled * (1 - pooled) * (1 / delivered_a + 1 / delivered_b))
16    z = (p_b - p_a) / se
17    p_value = 2 * (1 - NormalDist().cdf(abs(z)))
18    return p_a, p_b, z, p_value
19 
20 
21print(variant_for("919812345678", "hamper_copy_oct26"))
22p_a, p_b, z, p = two_proportion_test(402, 10_000, 517, 10_000)
23print(f"A {p_a:.2%}  B {p_b:.2%}  z={z:.2f}  p={p:.4f}")
24# A 4.02%  B 5.17%  z=3.88  p=0.0001

Assign arms by hashing the WhatsApp ID, then test clicks per delivered message — never per sent.

04 · Common mistakes, and the fix

Common mistakes with A/B testing

Testing a new template against a proven one
WhatsApp may apply template pacing to a new or unproven marketing template, holding part of its volume while early feedback comes in. If B is new and A is not, B's numbers arrive late and skewed. Let B finish pacing, or compare only delivered messages over the same window.
Calling the winner too early
Checking every hour and stopping as soon as B leads inflates false winners. Fix the sample size and decision window in advance and test significance once at the end.
Changing several things at once
New copy, a new image and a new button in one variant can tell you B won, but never why. Test one variable, or plan a larger multi-variant test with the sample to match.
Using read rate as the metric
Read receipts can be switched off by recipients, so read counts are understated and can differ between groups by chance. Prefer clicks, replies or orders, which the customer cannot hide.

05Example: Price-led or benefit-led copy for a Pune bakery chain

Before Diwali, a Pune bakery chain tests the first line of its hamper offer. Variant A leads with "Flat 20% off festive hampers"; variant B with "Free same-day delivery across Pune on festive hampers". Both use the same approved image and the same "Order now" URL button, and 20,000 opted-in contacts are split by hashing their WhatsApp IDs. After 48 hours A has 402 clickers from 10,000 deliveries (4.02%) and B 517 (5.17%) — a significant difference (p ≈ 0.0001). The chain sends B to the rest of its list and keeps a 5% holdout to check that the campaign adds orders at all.

06A/B testing: key facts and limits

Variants: 2 (A/B) or 3 (A/B/C) — more needs a bigger sample
Variables per test: One: copy, media, button text, offer or send time
Sample for 4% → 5% CTR: About 6,700 delivered per variant (95% / 80% power)
Denominator: Delivered messages, not sent
No new template needed: Header media and variable values
New template needed: Fixed body text, buttons or footer
Template review: Automatic, up to 24 hours

07Why does A/B testing matter?

Every broadcast is an opinion about what customers want — which offer, which picture, which button text. A/B testing replaces that opinion with evidence, one variable at a time, and because Meta charges per delivered marketing message, a variant that lifts orders per delivery lowers the real cost of every sale. Done badly — tiny samples, several changes at once, a winner called on day one — it produces confident decisions built on noise, which is worse than not testing.

08 · Tips from the bench

Best practices for A/B testing

Keep a running log of every test — hypothesis, sample, result — so the team stops re-testing settled questions.
Use utm_content=variant_a / variant_b on URL buttons so website analytics can see the split too.
Store template analytics daily: read and click data is only kept for 7 days after a send.
Treat a "no difference" result as an answer: keep the cheaper or simpler version.

09A/B Testing FAQs

How many contacts do I need for a WhatsApp A/B test?+

It depends on how big a difference you want to detect. Detecting a lift from a 4% to a 5% click-through rate needs about 6,700 delivered messages per variant at 95% confidence; smaller lifts need many more, so small lists should test bigger changes.

Do I need a new template for each A/B variant?+

Only if the fixed text changes. Header images or videos and variable values are filled in at send time, so they can be tested inside one approved template; different body text, buttons or footer need a second template and its approval.

Does WhatsApp have a built-in A/B testing feature?+

Not as a single switch on the Cloud API. You split the audience yourself, send each variant as a normal template message and compare Meta's per-template analytics or your own status and click data.

What is the difference between A/B testing and a holdout group?+

An A/B test compares two versions of a message; a holdout group receives nothing. A/B tells you which message works better, while a holdout tells you whether messaging adds sales that would not have happened anyway.

See InfiQ pricing

A/B Testing

Run the test: is variant B really better?

01What is A/B testing?

How does A/B testing work?

Write one hypothesis and one metric

Size the sample

Split the audience at random

Send both variants in the same window

Wait, then test significance once

03A/B testing in code

Common mistakes with A/B testing

Testing a new template against a proven one

Calling the winner too early

Changing several things at once

Using read rate as the metric