Chapter 1 · The scenario
Example: The 10:00 Diwali broadcast
A Pune marketplace lets sellers schedule Diwali offers, and most pick 10:00 sharp. At 10:00:00 the scheduler releases 600,000 sends to the workers at once. The sending number's throughput limit rejects most of them with error 130429, every worker retries after a fixed two seconds, and the same wall hits again at 10:00:02, 10:00:04 and onwards, while checkout OTPs sit stuck behind it. The fix combines four changes: the scheduler spreads each campaign's start across a randomised five-minute window, a shared token bucket paces sends just under the number's throughput, retries use full jitter, and OTPs get their own queue and priority.
Chapter 2 · What is the thundering herd problem?
The thundering herd problem happens when a large number of waiting processes, clients or jobs are released by the same event and rush a shared resource at once. The resource could serve them comfortably spread over time, but not all in the same second, so it slows down, times out or falls over — which often triggers another synchronised wave. The name comes from operating systems, where waking every thread blocked on one event caused exactly this stampede.
In messaging systems the herd has familiar shapes. An outage ends and thousands of queued sends retry on the same fixed schedule. Every worker's cached credential expires in the same minute and all of them refresh at once. A popular cache entry expires and every request misses together, a cache stampede. Dozens of campaigns are scheduled for exactly 10:00:00. Your webhook endpoint comes back after an hour of downtime and Meta's retried notifications arrive on top of live events.
Backoff jitter — randomising retry delays — is the best-known cure, but it only treats the retry case. The full toolkit adds client-side pacing so a sender never exceeds what the service can take, request coalescing so one refresh serves many waiters, staggered and randomised schedules, a gradual ramp-up after recovery, circuit breakers that let only a few probe requests through, and load shedding that rejects excess work quickly instead of failing slowly.
Key takeaways
- The thundering herd problem is many clients acting at the same instant against one resource, turning a recovery or a scheduled moment into an overload.
- Common triggers: synchronised retries after an outage, tokens or caches expiring together, and jobs scheduled for exactly 10:00:00.
- Jitter spreads retries in time, but a full cure also needs pacing, request coalescing, staggered schedules and load shedding.
- When your webhook endpoint recovers from downtime, Meta's retried notifications arrive alongside live traffic: size for the backlog.
- Watch per-second peaks, not per-minute averages: a herd hides comfortably inside a healthy-looking minute.
Chapter 3 · Why it matters
Why does the thundering herd problem matter?
A herd turns a short incident into a long one. The system recovers, gets knocked over by the rush of pent-up work, recovers and gets knocked over again, each cycle looking like a fresh outage. For WhatsApp senders it also wastes throughput: a wall of simultaneous sends meets the number's messages-per-second limit, collects 130429 errors and retries straight into the next wall. Designing for herds is what makes recovery fast and big broadcasts predictable.
Chapter 4 · How it unfolds
How does the thundering herd problem work?
Find the synchronising event
Look for anything that releases many actors together: a fixed retry delay, a shared expiry time, an on-the-hour schedule, a deploy that restarts every worker, a dependency coming back online.
Desynchronise it
Add random jitter to retry delays, cache TTLs, credential refreshes and scheduled jobs. Refresh a token at a random point before it expires, never at the expiry instant.
Coalesce duplicate work
Let one caller refresh a token or rebuild a cache entry while the others wait for its result (single-flight), instead of all of them doing the same work at once.
Admit only what fits
Pace sends with the token bucket algorithm below the number's throughput, ramp up gradually after recovery, and use a circuit breaker's half-open state to let a few requests test the water first.
Shed the rest cleanly
When demand still exceeds capacity, reject or defer low-priority work quickly — marketing before OTPs — rather than letting everything queue until it times out.
Chapter 5 · In code
The thundering herd problem in code
// Any expiring credential: a CRM OAuth token, a signed URL, a cached config.let cached = null; // { value, refreshAt }let inflight = null; // one refresh shared by every waiting callerexport async function getCredential() {if (cached && Date.now() < cached.refreshAt) return cached.value;inflight ??= fetchNewCredential().then(({ value, ttlMs }) => {const share = 0.6 + Math.random() * 0.2; // refresh at 60–80% of its lifetimecached = { value, refreshAt: Date.now() + ttlMs * share };return value;}).finally(() => { inflight = null; });return inflight;}
Two herd cures in one function: random refresh points stop workers expiring together, and the shared promise stops them refreshing together.
Chapter 6 · Where it goes wrong
Common mistakes with the thundering herd problem
Thinking jitter alone is enough
Jitter fixes synchronised retries, not a scheduler releasing 600,000 jobs at once or a cache expiring for everyone. Find every synchronising event and treat each one.
Restoring full traffic instantly
Opening the floodgates the moment a dependency recovers recreates the overload. Ramp concurrency up over minutes and let a circuit breaker's half-open probes confirm health first.
Watching per-minute averages
A minute averaging 70 sends per second can hide one second of 2,000 and many of almost nothing. Graph per-second peaks for anything with a per-second limit.
The same TTL everywhere
Caches and tokens created together with an identical lifetime expire together. Add a random spread to every TTL.
One queue for everything
When a herd of marketing sends fills the queue, OTPs and order updates wait behind it. Separate queues or priorities keep critical traffic moving.
Chapter 7 · The facts
The thundering herd problem: key facts and limits
| Classic triggers | Fixed-delay retries, shared expiry, on-the-hour schedules, mass restarts |
|---|---|
| Default Cloud API throughput | Up to 80 messages per second per number |
| Throughput error | 130429 when a number exceeds its throughput |
| Meta webhook retries | Up to 7 days, so a backlog lands when you recover |
| Full jitter delay | random(0, min(cap, base × 2^n)) |
| Cures beyond jitter | Coalescing, pacing, staggering, ramp-up, load shedding |
Chapter 8 · What good teams do
Best practices for the thundering herd problem
- Do not default campaign schedules to :00 and :30 — offer 10:07 as readily as 10:00.
- Size webhook consumers for backlog plus live traffic, not live traffic alone.
- Rehearse recovery: stop a dependency during a load test and watch what happens when it returns.
- Prefer full jitter to a small ±10% wobble, which barely breaks the synchrony.
Chapter 9 · Thundering Herd FAQs