Description
What is a circuit breaker?
A circuit breaker is a wrapper around calls to a dependency — the WhatsApp Cloud API, a CRM, a payment gateway, a Flow data endpoint — that watches the results and stops making calls once failures pass a threshold. Like its electrical namesake it trips to protect the rest of the system: while open, calls fail immediately without touching the dependency, so workers are not tied up waiting on timeouts and the struggling service gets room to recover.
The pattern has three states. Closed is normal: calls pass through and the breaker counts failures over a rolling window, for example the last 20 calls. When the failure rate crosses the threshold — say 50% with at least 20 calls — it moves to open and rejects calls for a cool-down period such as 30 seconds. Then it turns half-open and allows a few trial calls: if they succeed it closes, and if one fails it opens again for another cool-down.
A circuit breaker handles sustained failure; the Retry Policy handles individual transient errors. Without a breaker, every request during a dependency outage runs its full retry schedule, multiplying load exactly when the dependency can least take it. Breakers also pair naturally with Graceful Degradation — when the CRM breaker is open, the bot answers from cached order data instead of timing out — and with the Bulkhead Pattern, which caps how many workers a slow dependency can hold while the breaker is still counting.
Key takeaways
- A circuit breaker stops calling a failing dependency after an error threshold, fails fast while open, and probes for recovery.
- It has three states: closed (calls flow), open (calls rejected instantly) and half-open (a few trial calls test recovery).
- Retries handle individual transient errors; a circuit breaker handles sustained failure, so the two work together.
- Count only failures that reflect the dependency's health — timeouts, 5xx, Graph service errors — not bad-input errors like 132000.
- Scope breakers per dependency and, for account-level errors, per phone number, and pair the open state with a fallback.
Mechanics
How does a circuit breaker work?
step 1 — Wrap each dependency
Put one breaker in front of each external service. For errors that belong to a single account — a number in maintenance, a locked account — key the breaker by
phone_number_id, so one bad number does not block the rest.step 2 — Classify every result
Sort results using Meta's API error codes. Count timeouts (a Request Timeout on every call turns a hang into a countable failure), connection errors, HTTP 5xx and Graph service errors (codes 1, 2, 131000, 131016) as failures. Do not count caller mistakes such as Error 132000 (Parameter Count Mismatch) or 131026 (message undeliverable): they say nothing about the dependency's health.
step 3 — Trip on a rate over a minimum volume
Open when, for example, at least 50% of the last 20 calls failed. The minimum volume stops one failed call at 3 am from tripping the breaker.
step 4 — Fail fast and fall back while open
Reject calls immediately with a clear error, park sends in the queue, serve cached data or a simpler reply, and emit a metric so alerting can see the open state.
step 5 — Probe in half-open
After the cool-down, let a few trial calls through. Close when they succeed; reopen on the first failure, optionally with a longer cool-down each time. Some implementations probe a Health Check endpoint instead, but real calls prove more.
Usage
A circuit breaker in code
type State = "closed" | "open" | "half-open";export class CircuitBreaker {private state: State = "closed";private results: boolean[] = []; // rolling window, true = failureprivate openedAt = 0;private trialsStarted = 0;private trialsPassed = 0;constructor(private readonly opts = { window: 20, minCalls: 20, failureRate: 0.5, coolDownMs: 30_000, trialCalls: 3 },) {}async call<T>(fn: () => Promise<T>, isFailure: (err: unknown) => boolean): Promise<T> {if (this.state === "open") {if (Date.now() - this.openedAt < this.opts.coolDownMs) throw new Error("circuit open: failing fast");this.state = "half-open";this.trialsStarted = 0;this.trialsPassed = 0;}if (this.state === "half-open") {if (this.trialsStarted >= this.opts.trialCalls) throw new Error("circuit half-open: trials in flight");this.trialsStarted++;}try {const value = await fn();this.record(false);return value;} catch (err) {this.record(isFailure(err)); // caller errors do not count against the dependencythrow err;}}private record(failed: boolean) {if (this.state === "half-open") {if (failed) return this.trip();if (++this.trialsPassed >= this.opts.trialCalls) {this.state = "closed";this.results = [];}return;}this.results = [...this.results, failed].slice(-this.opts.window);const failures = this.results.filter(Boolean).length;if (this.results.length >= this.opts.minCalls && failures / this.results.length >= this.opts.failureRate) {this.trip();}}private trip() {this.state = "open";this.openedAt = Date.now();this.results = [];}}
Closed counts failures in a rolling window, open fails fast for the cool-down, half-open lets three trial calls decide.
Example
Example: A CRM outage at a Pune grocery app
A grocery delivery app in Pune looks up each customer's last order in its CRM before the WhatsApp bot answers "Where is my order?". One evening the CRM starts timing out after 10 seconds. Without a breaker, 40 workers spend all their time waiting and inbound messages queue for 12 minutes. With a breaker — open at 50% failures over 20 calls, a 30-second cool-down and three trial calls — it trips within seconds. The bot answers from the order status cached in Redis, adding that live tracking is delayed, and the breaker closes by itself when the CRM recovers 18 minutes later.
Properties
A circuit breaker: key facts and limits
- States
- Closed, open, half-open
- Example trip rule
- ≥ 50% failures over ≥ 20 calls (illustrative)
- Example cool-down
- 30 s, growing on repeated trips
- Count as failures
- Timeouts, HTTP 5xx, Graph codes 1, 2, 131000, 131016
- Do not count
- Caller errors such as 132000, 131026, 131047
- Typical scope
- Per dependency; per phone_number_id for account errors
Caveats
Common mistakes with a circuit breaker
Counting client errors as failures
A batch of sends with a bad template parameter (132000) or to unreachable numbers (131026) can trip a breaker on a perfectly healthy Cloud API. Count only errors that reflect the dependency's health.
One global breaker for everything
If a single breaker guards the Cloud API for every number and tenant, one number in maintenance blocks them all. Scope breakers per dependency and, where errors are account-specific, per phone number.
Opening with no fallback
A breaker that just throws turns a slow failure into a fast one — still a failure. Decide what happens while open: queue and retry later, serve cached data, or degrade the reply.
No minimum call volume
With a 50% threshold and no minimum, one failure out of two calls at night trips the breaker. Require a minimum number of calls in the window before the rate is evaluated.
Invisible state changes
If nobody knows the breaker opened, a dependency outage looks like a quiet queue. Emit a metric and a log line on every transition, and alert when a breaker stays open.
Why it matters
Why does a circuit breaker matter?
During an outage the expensive part is often not the failure but the waiting. If a CRM hangs for 30 seconds per call and every webhook handler calls it, a few hundred events tie up every worker, the queue backs up, webhook acknowledgements slow until Meta starts retrying, and inbound messages for unrelated customers stall. A breaker turns 30-second hangs into instant, handled failures, keeps workers free for everything else, and stops your own retries from hammering a service that is trying to come back.
Notes
Best practices for a circuit breaker
- Check the breaker before the whole retry loop, not inside each attempt, so an open circuit stops the schedule at once.
- Lengthen the cool-down on repeated trips (30 s, 60 s, 120 s) to avoid flapping during a long outage.
- Do not trip on 130429: throughput errors call for pacing and backpressure, not for cutting the dependency off.
- Give operators a manual trip and reset for planned maintenance on partner systems.
Questions
Circuit Breaker FAQs
- What is the difference between a circuit breaker and a retry?
- A retry repeats one failed call in the hope the error was transient; a circuit breaker stops all calls once failures are sustained. Use both: retries with exponential backoff for blips, and a breaker so retries stop when the dependency is clearly down.
- What happens in the half-open state?
- The breaker lets a small number of trial calls through to test whether the dependency has recovered. If they succeed it closes and normal traffic resumes; if one fails it opens again for another cool-down.
- Should I put a circuit breaker in front of the WhatsApp Cloud API?
- Yes, for service-level failures such as timeouts, 5xx responses and Graph service errors. Do not trip it on caller errors like 132000, and handle throughput errors (130429) with pacing rather than by opening the circuit.
- What thresholds should a circuit breaker use?
- A common starting point is opening at 50% failures over at least 20 calls, a 30-second cool-down and three trial calls. Tune from real traffic: low-volume dependencies need longer windows, critical paths a faster trip.