Description
What is consumer lag?
Consumer lag is the gap between what producers have written to a queue or stream and what consumers have processed. It can be counted in messages — queue depth, or in a partitioned log the difference between the latest offset and the consumer group's committed offset — or in time, as the age of the oldest message still waiting. Either way it answers one operational question: are we keeping up?
The two forms tell different stories. Depth depends on traffic: 50,000 waiting sends is normal halfway through a broadcast paced at 80 messages per second, but alarming on an OTP queue. Age does not: if the oldest inbound webhook event has waited 90 seconds, a customer's question has gone unanswered for 90 seconds, whatever the volume. Combining depth with the processing rate gives the most useful forecast — drain time, roughly the backlog divided by the net rate at which it shrinks.
Consumer lag is the backlog metric for a Message Queue; P99 Latency measures individual requests. The two can disagree: each job may finish in 200 ms while the queue holds an hour of work because too few workers are running. Lag rises when producers outrun consumers — a broadcast enqueued in one burst, a webhook spike after an outage, crashed workers, a slow dependency — and it is usually the earliest sign that Backpressure, Horizontal Scaling or load shedding is needed.
Key takeaways
- Consumer lag is how far processing trails production: the messages waiting, or how old the oldest waiting message is.
- Age of the oldest message is the better alert signal, because it measures what customers feel — delay — at any traffic level.
- Drain time ≈ backlog ÷ (consume rate − produce rate); if that net rate is zero or negative, the backlog never clears.
- Rising lag on inbound webhooks means slow replies to customers; rising lag on sends means a broadcast will finish late.
- Lag measures the queue as a whole while p99 latency measures single requests — each job can be fast while lag grows.
Properties
Consumer lag: key facts and limits
- Depth in a partitioned log
- Log-end offset − committed offset, per partition
- Lag as time
- Now − enqueue time of the oldest waiting message
- Drain time
- Backlog ÷ (consume rate − produce rate)
- One number at default throughput
- 80 msg/s ≈ 288,000 messages per hour
- Example inbound alert
- Oldest event > 2 min for 5 min (illustrative)
- Not the same as
- p99 latency, which times single requests
Mechanics
How does consumer lag work?
step 1 — Measure depth per queue or partition
Read waiting-message counts from the broker, or in a partitioned log compute log-end offset minus committed offset for each partition and sum across the consumer group.
step 2 — Measure the age of the oldest message
Stamp each job with its enqueue time and export now minus the oldest waiting timestamp. Many brokers expose this directly as a metric.
step 3 — Turn it into drain time
Drain time ≈ backlog ÷ (consume rate − produce rate). 216,000 waiting sends at a net 60 per second is an hour; at a net zero it is never.
step 4 — Alert on age and trend
Page when the oldest inbound webhook event has been waiting more than, say, 2 minutes for 5 minutes; open a ticket when a broadcast's forecast finish time slips past its deadline.
step 5 — Act on the cause, not the symptom
If workers are the bottleneck, scale out. If a dependency or the number's throughput is, add backpressure upstream or shed low-priority work — more workers will not help.
Usage
Consumer lag in code
# Lag per partition for the webhook consumer group (some columns trimmed)kafka-consumer-groups.sh --bootstrap-server kafka:9092 \--describe --group wa-webhook-processor# GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID# wa-webhook-processor wa.inbound 0 1840221 1840230 9 worker-1-7c1f# wa-webhook-processor wa.inbound 1 1838904 1851322 12418 worker-2-a93e# wa-webhook-processor wa.inbound 2 1841177 1841181 4 worker-3-02bd# One partition far behind while the others keep up: a stuck or slow consumer,# not a traffic spike. Adding workers will not fix it; restarting worker-2 might.
LAG is log-end offset minus committed offset. Read it per partition — the sum hides a single stuck consumer.
Why it matters
Why does consumer lag matter?
Lag predicts customer pain before customers report it. Support wants to know inbound messages reach agents within seconds; marketing wants to know whether the 7 pm offer will land at 7 pm or 9 pm; the on-call engineer wants to know whether last night's webhook backlog is draining or growing. Error rates stay flat through all of these — nothing fails, it just waits — while lag shows them at once; among the Four Golden Signals, it is how a queue reports saturation. That is why it belongs on every queue's dashboard and in its alerting.
Example
Example: An evening webhook backlog at a Bhopal bank
A cooperative bank in Bhopal processes inbound WhatsApp messages and delivery statuses through a queue with six workers. At 7:05 pm a deploy leaves two of them crash-looping. Error rates stay at zero, but the oldest inbound message ages from 3 seconds to 4 minutes in 20 minutes, and the lag alert pages the on-call engineer. The drain-time panel shows a negative net drain rate: four workers finish less than arrives, so the backlog would never clear on its own. Restarting the workers and temporarily scaling to twelve brings the oldest message back under 10 seconds by 7:40 pm, before the support team hears from a single customer.
Caveats
Common mistakes with consumer lag
Alerting on depth alone
A fixed depth threshold either fires through every healthy broadcast or misses a small but stuck OTP queue. Alert on the oldest message's age, and tune any depth thresholds per queue.
Averaging across partitions or queues
A total lag of 12,000 can hide one partition at 12,000 and five at zero — a stuck consumer, not a load problem. Break lag down per partition, queue and consumer.
Scaling when the bottleneck is downstream
If sends are capped by the number's throughput, more workers raise 130429 errors, not drain speed. Check what limits the consume rate before adding capacity.
Watching only your own queue
Your queue can show zero lag while Meta is retrying webhooks your endpoint rejected. Track webhook acknowledgement failures, and compare event timestamps with arrival times.
Notes
Best practices for consumer lag
- Stamp every job with its enqueue time; time-based lag is impossible without it.
- Show a forecast finish time on broadcast dashboards, so marketers see slippage before the send window closes.
- Compare a webhook message's
timestampwith its arrival time to spot delays that happen before your queue. - Autoscale consumers on lag rather than CPU: an idle-looking worker fleet can be hopelessly behind.
Questions
Consumer Lag FAQs
- What is a good consumer lag?
- Good lag stays within your latency objective and returns to near zero after spikes. For inbound customer messages that means seconds; for a paced broadcast it means a large but steadily shrinking backlog that finishes on schedule.
- Should I measure consumer lag in messages or in time?
- Measure both, but alert on time. Message counts depend on traffic volume, while the age of the oldest waiting message measures directly how long work — and customers — have been waiting.
- How do I estimate when a WhatsApp broadcast will finish?
- Divide the remaining backlog by the net send rate. At the default 80 messages per second one number sends at most about 288,000 messages an hour, so 500,000 still queued needs well over an hour and a half.
- What is the difference between consumer lag and latency?
- Latency measures how long one request or job takes; consumer lag measures how much work is waiting. Every job can be fast while lag grows, because jobs arrive faster than the workers can take them.