Description
What is the bulkhead pattern?
The bulkhead pattern takes its name from the watertight compartments in a ship's hull: if one floods, the bulkheads keep the water out of the rest. In software it means partitioning a shared resource — worker threads, queue consumers, database connections, outbound rate budget — into fixed pools, each serving one class of work. When one class misbehaves, it can exhaust only its own pool; everything else keeps its capacity.
On a WhatsApp platform the natural compartments are traffic types and tenants. OTPs, order updates, marketing broadcasts and inbound webhook processing each get their own Message Queue and worker pool, so a 500,000-message campaign cannot delay a login code. A multi-tenant provider adds per-tenant concurrency caps so one customer's runaway import does not slow everyone, and per-dependency pools so a hanging CRM integration holds only its own ten connections instead of every worker in the fleet.
Bulkheads isolate capacity up front, before anything fails; a Circuit Breaker reacts once failures occur, by cutting off a dependency that keeps erroring. They complement each other: the bulkhead limits how much damage a slow dependency can do while the breaker is still counting errors. Mind the platform's own boundaries too. Throughput is set per business phone number, but Messaging Limits & Tiers apply to the whole business portfolio: the count of unique users reached in a rolling 24 hours is shared by all its numbers — so separate numbers are a bulkhead for send rate, not for reach.
Key takeaways
- The bulkhead pattern partitions capacity so a flood or failure in one compartment cannot use up the resources of the others.
- Typical bulkheads are separate queues, worker pools, connection pools and rate budgets per traffic type, tenant or phone number.
- A bulkhead isolates capacity up front; a circuit breaker reacts after failures start. Most systems need both.
- Throughput is set per phone number, but messaging limits are shared across a business portfolio — splitting numbers isolates one, not the other.
- Size each compartment for its own peak, and give critical traffic such as OTPs reserved capacity nothing else can borrow.
Properties
The bulkhead pattern: key facts and limits
- Name origin
- Watertight compartments in a ship's hull
- Common compartments
- Queues, worker pools, connection pools, rate budgets
- Throughput scope
- Per business phone number (80 msg/s by default)
- Messaging-limit scope
- Per business portfolio, shared by all its numbers
- Example OTP reservation
- 20 of 80 msg/s (illustrative)
- Reacts to failures?
- No — it isolates up front; a circuit breaker reacts
Usage
The bulkhead pattern in code
import { Worker } from "bullmq";const connection = { host: "redis", port: 6379 };// One queue and worker pool per compartment. concurrency caps parallel jobs;// limiter caps each lane's share of the number's 80 msg/s (total 75, headroom 5).new Worker("wa-otp", sendJob, {connection,concurrency: 4,limiter: { max: 20, duration: 1000 }, // reserved: 20 msg/s for OTPs});new Worker("wa-transactional", sendJob, {connection,concurrency: 8,limiter: { max: 30, duration: 1000 },});new Worker("wa-marketing", sendJob, {connection,concurrency: 8,limiter: { max: 25, duration: 1000 }, // a campaign can never take more});
Each lane has its own backlog, concurrency and rate cap, so a broadcast cannot eat into the OTP slice.
Mechanics
How does the bulkhead pattern work?
step 1 — Identify the compartments
Group work by business criticality and failure domain: authentication, transactional updates, marketing, inbound processing and each external integration. Multi-tenant systems add a dimension per tenant or per Phone Number ID.
step 2 — Give each its own queue and worker pool
Separate queues with their own consumers and concurrency limits mean a backlog in one lane never sits in front of another lane's jobs.
step 3 — Split the shared rate budget
A number's throughput, 80 messages per second by default, is shared by everything sent from it. Reserve a slice for OTPs — say 20 per second — and cap the other lanes so their total stays below the limit.
step 4 — Bound calls to each dependency
Wrap each external service in a semaphore with a maximum concurrency and a small wait queue. When it is full, reject fast instead of letting callers pile up behind a hung dependency.
step 5 — Monitor every compartment
Track queue age, saturation and rejections per pool. A full bulkhead is a signal to investigate that workload, not to quietly raise the limit for everyone.
Why it matters
Why does the bulkhead pattern matter?
Shared pools fail together. One tenant's webhook handler that calls a slow ERP can tie up every worker in a shared pool within seconds, and suddenly OTPs, replies and statuses for every tenant queue behind it. Bulkheads turn that platform-wide outage into a single degraded compartment — the difference between one unhappy customer and a status-page incident. They also make capacity planning honest, because each workload's budget is written down instead of assumed.
Caveats
Common mistakes with the bulkhead pattern
Bulkheads that share a hidden pool
Separate queues served by the same worker process, the same database connection pool or an unlimited Redis client are one compartment in disguise. Trace every resource each lane consumes.
Splitting numbers to escape messaging limits
Throughput is per phone number, but messaging limits are counted across the business portfolio and shared by all its numbers. Extra numbers isolate send rate, not how many unique users the portfolio can reach.
Pools sized for averages
A compartment sized for its average load saturates at every peak. Size each for its own peak, and let low-priority lanes borrow idle capacity only if they can give it back instantly.
Too many tiny compartments
One pool per tenant with hundreds of tenants leaves most capacity idle while the busiest tenant waits. Group small tenants into shared tiers and isolate only the largest or riskiest.
Example
Example: One shared pool at a Kochi travel portal
A travel portal in Kochi ran every WhatsApp job — booking confirmations, OTPs, marketing and a sync to a partner hotel system — on one pool of 32 workers. When the hotel system started taking 30 seconds per call, all 32 workers were stuck on it within a minute and OTP sends stalled for 25 minutes. The rebuild gives OTPs their own queue, 4 workers and a reserved 20 messages per second; the hotel sync gets 6 workers behind a 10-call semaphore with a 5-second timeout. The next time the hotel system slowed down, only its sync fell behind, and customers kept logging in.
Notes
Best practices for the bulkhead pattern
- Start with three lanes — authentication, transactional and marketing — before slicing by tenant.
- Put a timeout on every call inside a bulkhead; a semaphore without one just queues the hang more politely.
- Show per-compartment queue age, active jobs and rejections on one dashboard, so a full lane is obvious.
- Combine with a circuit breaker per dependency: the bulkhead caps the damage while the breaker decides to stop calling.
- Bulkheads decide who gets capacity; Horizontal Scaling adds more of it, Backpressure slows producers, and Load Shedding decides what to drop when a lane stays full.
Questions
Bulkhead Pattern FAQs
- What is the difference between a bulkhead and a circuit breaker?
- A bulkhead limits how much capacity a workload can use before anything fails; a circuit breaker stops calls after failures pass a threshold. The bulkhead contains a slow dependency immediately, and the breaker then cuts it off until it recovers.
- Should each WhatsApp phone number have its own queue?
- Often yes, because throughput is set per number and one number's 130429 errors should slow only its own sends. Messaging limits are shared across the business portfolio, though, so per-number queues do not isolate those.
- How do I size a bulkhead?
- Size each compartment for its own peak plus headroom, using measured job duration and arrival rate. Reserve capacity for critical traffic such as OTPs, and keep the total of all lanes below the shared limit — for example 75 of 80 messages per second.
- Does the bulkhead pattern waste capacity?
- Some, by design: reserved capacity sits idle when its lane is quiet. That idle slice is the insurance that keeps OTPs flowing during a broadcast, so keep it small and size it from real peaks.