Chapter 1 · The scenario
Example: Seventy seconds at an Ahmedabad pharmacy chain
At 10:02 the primary database behind a pharmacy chain's order-update service stops responding. Health checks fail three times in six seconds; the orchestrator fences the old node and promotes the replica in the second zone by 10:02:20, and the stable database endpoint points at it fifteen seconds later. Workers had paused sending when writes failed, so no messages were lost, and the webhook receiver returned 503 for about a minute, which Meta's retries covered. At 10:30 the team reconciles eleven sends that timed out mid-switch: nine already had a sent status by wamid, two are re-sent. Failback is scheduled for 2 am.
Chapter 2 · What is failover?
Failover is the automatic switch from a failed component to a redundant one: a database replica promoted to primary, traffic moved from a dead application instance to a healthy one, or a whole zone's workload shifted to another. The goal is that a hardware fault, a crashed process or a zone outage costs seconds of disruption rather than hours, without anyone being woken up to make the switch by hand.
Every failover has the same phases. Detection: health check probes or heartbeats notice the primary is gone, usually after several consecutive failures so one slow response does not trigger a switch. Decision and fencing: the system confirms the primary is really down and cuts it off so it cannot keep accepting writes. Promotion and redirection: the standby takes over, and clients find it through a stable DNS name, a virtual IP or a load balancing rule. Recovery: queued work resumes, and later the team plans failback, the deliberate switch back or the rebuild of the old primary as the new standby.
Setups come in two shapes. In active-passive, a standby sits idle (or serves only reads) until it is promoted; it is simpler but the switch takes time and the standby's capacity is unused. In active-active, every copy serves live traffic and failover just means routing around the one that died; it is faster but needs stateless services or carefully replicated data.
Failover is one tool of high availability (HA), and narrower than disaster recovery (DR), which plans for losing a whole region or data store. It also differs from a fallback channel: failover switches your own servers, while a fallback channel re-sends a customer's message by SMS or another channel. Meta operates the Cloud API itself, so its incidents call for queueing and graceful degradation rather than failover.
Key takeaways
- Failover switches traffic from a failed primary to a standby automatically, so a component failure becomes a short blip instead of an outage.
- Detection time is roughly the health-check interval times the failure threshold: 2 seconds x 3 failures is about 6 seconds.
- Active-passive keeps an idle standby ready; active-active runs every copy live and simply stops routing to the one that fails.
- Fence the old primary before promoting a new one, or two primaries accept writes at once (split brain).
- Failover switches your infrastructure; a fallback channel switches the customer to SMS, and neither can fail over Meta's Cloud API.
Chapter 3 · How it unfolds
How does failover work?
Run a standby that is truly ready
Keep a replica in another zone with replicated data, the same code version, and every secret it needs: the system user access token, app secret and webhook verify token.
Detect failure with health checks
Probe the primary every few seconds and require several consecutive failures. Detection time is roughly interval x threshold, so 2 seconds x 3 is about 6 seconds.
Fence the old primary, then promote
Cut the failed primary off (revoke its lease, stop the instance, block its writes) before promoting the standby, so the two can never both accept writes.
Redirect clients through a stable name
Clients and Meta's webhook deliveries should reach a stable hostname, virtual IP or load balancer, never an instance address. Short DNS TTLs and connection-pool refreshes let clients move within seconds.
Recover in-flight work, then plan failback
Resume queued sends, reconcile any that timed out against status webhooks before resending, and later rebuild the old primary as the new standby or switch back in a quiet window.
Chapter 4 · Why it matters
Why does failover matter?
Components fail constantly at scale: disks, virtual machines, zones, deploys. If recovering from each one needs a human, every failure becomes an outage measured in tens of minutes, and an OTP or order-update service misses its SLO with every incident. Automatic, rehearsed failover turns those events into seconds of elevated latency. For a WhatsApp backend it also protects the parts Meta cannot retry for you: the outbound sends that would otherwise be lost or doubled mid-switch.
Chapter 5 · In code
Failover in code
backend webhook_receiversoption httpchkhttp-check send meth GET uri /readyzhttp-check expect status 200# probe every 2 s; 3 failures = down (~6 s detection), 2 passes = back updefault-server inter 2s fall 3 rise 2server zone_a 10.0.1.10:8080 checkserver zone_b 10.0.2.10:8080 check backup # only used when zone_a is down
The backup server only receives traffic when every primary server has failed its health checks.
Chapter 6 · The facts
Failover: key facts and limits
| Detection time | About check interval x failure threshold (2 s x 3 = 6 s) |
|---|---|
| Active-passive | Idle or read-only standby, promoted on failure |
| Active-active | All copies live; failed one is routed around |
| Split-brain guard | Fence the old primary before promotion |
| Meta webhook retries | Up to 7 days, so a brief receiver switch loses nothing |
| Failback | A second, planned switch, not an automatic reflex |
Chapter 7 · Where it goes wrong
Common mistakes with failover
Split brain
If the old primary is only unreachable, not dead, it may keep accepting writes after the standby is promoted, and the two diverge. Use a consensus or lease-based manager and fence the old node before promotion.
A standby missing configuration
A replica without the current access token, app secret or environment variables comes up and immediately fails every send. Deploy config and secrets to both sides from the same source and check them in drills.
Long DNS TTLs and sticky connections
Clients that cached the old address for an hour, or connection pools that never reconnect, keep hammering the dead primary. Keep TTLs short on failover names and make pools validate connections.
Duplicates from in-flight sends
A Graph API call that timed out during the switch may still have been delivered. Record send attempts with an Idempotency Key and reconcile against wamids and status webhooks before resending.
Never rehearsing
A failover path that has never run usually fails the first time it is needed. Trigger it on purpose in a quiet window every quarter and time each phase.
Chapter 8 · Failover FAQs
Failover FAQs
What is the difference between failover and fallback?+
Active-passive or active-active: which do I need?+
How long does failover take?+
Can I fail over the WhatsApp Cloud API itself?+
What is failback?+
Chapter 9 · What good teams do
Best practices for failover
- Register a stable hostname as your webhook callback URL so a failover never means re-subscribing with Meta.
- Alert on every automatic failover even when it succeeds; frequent switches point to a real problem.
- Size the standby to carry full load, not just to boot.
- Write the failback runbook before you need it; it is usually the riskier half.