Alerting: Burn-Rate Alerts for Messaging Systems

01 · How it works, in 5 steps

How does alerting work?

1
Start from the SLO, not the dashboard
Pick a Service Level Indicator customers feel — share of sends accepted by the Cloud API within 2 seconds, share of webhooks acknowledged with 200, age of the oldest queued message — and set a target and window for each, for example 99.9% over 30 days.
2
Compute the burn rate
Burn rate = error ratio ÷ (1 − SLO). At 99.9%, an error ratio of 0.1% burns at 1×, 0.6% at 6× and 1.44% at 14.4×. Record the ratio for each window as its own metric so the alert rules stay readable.
3
Combine windows and thresholds
Page when both the 1-hour and 5-minute burn exceed 14.4×, or both the 6-hour and 30-minute burn exceed 6×; open a ticket when the 3-day and 6-hour burn exceed 1×. The long window proves the problem is significant, the short one lets the alert clear soon after recovery.
4
Add the alerts an SLO misses
Cover silence and saturation: webhook events per minute dropping to zero, Consumer Lag above a threshold, a template moving to PAUSED, a phone number's quality rating changing, and a Synthetic Monitoring probe that sends a test message end to end every few minutes. Several of these arrive as webhooks, not as error ratios.
5
Route, group and attach a runbook
Pages go to the on-call rotation and tickets to the team queue; related alerts are grouped so one outage is one page. Every alert carries a summary, a dashboard link and a runbook URL — and any alert that fires without a useful action is tuned or deleted.

02Try it

Hands-on: use the controls to work through it yourself.

Where is the needle: page, ticket or sleep?

5 / 5

Within budget0.0–1.0 ×
Ticket if sustained1.0–6.0 ×
Slow-burn page6.0–14.4 ×
Fast-burn page14.4–20.0 ×

Reading 5 of 5Within budget

Rolled back

0.6 ×

Errors stop. The 5-minute window falls below the threshold within minutes and the page resolves, even though the 1-hour average is still elevated.

The needle is the send error budget's burn rate against a 99.9% SLO. Step through five readings, from a normal weekday to a bad deploy and its rollback, to see which zone pages, which opens a ticket and which stays quiet.

03What is alerting?

Alerting is the part of monitoring that decides when a human must be told something is wrong, how urgently, and who that human is. It turns metrics — error ratios, latency percentiles, queue age, webhook rates — into two kinds of notification: a page, which interrupts the on-call engineer now because customers are being hurt, and a ticket, which can wait for working hours. Good alerting is judged on precision (does every page mean real impact?) and recall (does every real incident page someone?).

Modern practice alerts on symptoms — usually drawn from the Four Golden Signals of latency, traffic, errors and saturation — measured against a Service Level Objective rather than on causes. Instead of "CPU above 80%", the rule asks how fast the error budget is being spent. The burn rate is the observed error ratio divided by the ratio the SLO allows: with a 99.9% target the budget is 0.1%, so a 1.44% error ratio is a burn rate of 14.4 — fast enough to spend 2% of a 30-day budget in one hour. Multi-window, multi-burn-rate rules page on fast burns, open tickets on slow ones, and require a short window to agree so the alert resets quickly once the problem stops.

On a WhatsApp Business API integration the customer-facing symptoms are specific: sends failing or slowing (Cloud API errors such as 130429, or timeouts), inbound and status webhooks not arriving, queues aging, templates being paused. Alerting raises the signal; incident management is what the team does once it fires. An alert nobody can act on is noise, however accurate it is.

Key takeaways

Alerting turns metrics into pages and tickets that fire when customers are hurt, not whenever a number twitches.
Burn rate is the error ratio divided by the budget the SLO allows (1 − target); a burn rate of 1 spends the budget exactly over the window.
A common 30-day setup pages at 14.4× burn over 1 hour (2% of budget) or 6× over 6 hours (5%), and opens a ticket at 1× over 3 days.
Pair every long window with a short one, such as 1 hour with 5 minutes, so an alert clears soon after the problem stops.
For WhatsApp integrations, alert on silence too: a webhook rate that drops to zero is an outage that produces no errors.

04Example: A bad deploy at a Hyderabad diagnostics chain

A diagnostics chain in Hyderabad sends report-ready utility templates and OTPs through the Cloud API, with a 99.9% 30-day SLO on sends. On a Saturday evening a deploy breaks a template parameter for one report type, and about 15% of all sends start failing with error 132000. The burn rate jumps to roughly 150×: the 5-minute window crosses 14.4× almost at once and the 1-hour window about six minutes later, so the page fires within minutes. The runbook points at the last deploy, the on-call engineer rolls it back, and the short window drops below the threshold minutes later, resolving the page. A year earlier the team's only alert was CPU above 80%, and they heard about failures like this on Monday from customers.

05 · In code

Alerting in code

1# SLO: 99.9% of WhatsApp sends accepted over 30 days (error budget = 0.001)
2groups:
3  - name: whatsapp-send-slo
4    rules:
5      # Error ratio per window, recorded once and reused by every alert
6      - record: wa_send:error_ratio:rate5m
7        expr: sum(rate(wa_send_total{result="error"}[5m])) / sum(rate(wa_send_total[5m]))
8      - record: wa_send:error_ratio:rate30m
9        expr: sum(rate(wa_send_total{result="error"}[30m])) / sum(rate(wa_send_total[30m]))
10      - record: wa_send:error_ratio:rate1h
11        expr: sum(rate(wa_send_total{result="error"}[1h])) / sum(rate(wa_send_total[1h]))
12      - record: wa_send:error_ratio:rate6h
13        expr: sum(rate(wa_send_total{result="error"}[6h])) / sum(rate(wa_send_total[6h]))
14      - record: wa_send:error_ratio:rate3d
15        expr: sum(rate(wa_send_total{result="error"}[3d])) / sum(rate(wa_send_total[3d]))
16 
17      - alert: WhatsAppSendBudgetFastBurn      # 2% of the budget in 1 hour
18        expr: |
19          wa_send:error_ratio:rate1h > (14.4 * 0.001)
20          and wa_send:error_ratio:rate5m > (14.4 * 0.001)
21          and sum(increase(wa_send_total[1h])) > 100   # tiny traffic cannot page
22        labels: { severity: page }
23        annotations:
24          summary: "WhatsApp sends burning error budget at over 14.4x"
25          runbook_url: "https://runbooks.example.in/whatsapp/send-errors"
26 
27      - alert: WhatsAppSendBudgetSlowBurn      # 5% of the budget in 6 hours
28        expr: |
29          wa_send:error_ratio:rate6h > (6 * 0.001)
30          and wa_send:error_ratio:rate30m > (6 * 0.001)
31        labels: { severity: page }
32 
33      - alert: WhatsAppSendBudgetTicket        # 10% of the budget in 3 days
34        expr: |
35          wa_send:error_ratio:rate3d > 0.001
36          and wa_send:error_ratio:rate6h > 0.001
37        labels: { severity: ticket }
38 
39      - alert: WhatsAppWebhooksSilent          # silence is an outage with no errors
40        # "or vector(0)" makes the rule fire even when the series vanishes entirely
41        expr: (sum(rate(wa_webhook_events_total[15m])) or vector(0)) == 0
42        for: 10m
43        labels: { severity: page }

Multi-window, multi-burn-rate rules for a 99.9% send SLO, plus a silence alert for webhooks that stop arriving.

06 · Key facts

Alerting: key facts and limits

Burn rate: Error ratio ÷ (1 − SLO target)
Fast-burn page: 14.4× over 1 h and 5 min (2% of a 30-day budget)
Slow-burn page: 6× over 6 h and 30 min (5% of budget)
Ticket: 1× over 3 days and 6 h (10% of budget)
Time to fire: ≈ long window × threshold ÷ actual burn rate
Budget at 99.9% over 30 days: About 43 minutes of total outage

07 · Why it matters

Why does alerting matter?

Every team meets two alerting failures: the outage nobody was paged for, and the pager that cries wolf until it is muted. Both cost customers. A broadcast that silently stalls or a webhook endpoint returning 500s for an hour means OTPs not delivered and replies never seen; a channel with 40 non-actionable alerts a day trains engineers to ignore the 41st. Burn-rate alerting ties the pager to what customers experience, so an engineer woken at 3 am knows the problem is real, how fast it is spreading and which runbook to open.

08 · Mistakes to avoid

Common mistakes with alerting

Alerting on causes instead of symptoms
CPU, memory and pod restarts rarely map to customer pain and fire constantly. Page on SLO burn and customer-visible symptoms; keep cause metrics on dashboards, where they help diagnosis once someone is looking.
Ratio alerts on tiny traffic
At 3 am, one failed send out of 20 is a 5% error ratio. Add a minimum request count to the rule, or use longer windows for low-volume flows, so a small denominator cannot page anyone.
No alert for silence
A webhook endpoint that stops receiving events produces no errors at all. Alert on the absence of expected traffic — inbound and status webhooks per minute, sends per minute during a scheduled broadcast.
Every alert is a page
When tickets and pages share one channel, engineers mute it. Reserve pages for fast budget burn and outages; anything slow-moving becomes a ticket reviewed in working hours.
Alerts without a runbook
"Error rate high" leaves the responder guessing. Say what is affected, link the dashboard and recent deploys, and list the first three things to check.

09 · Pro tips

Best practices for alerting

Review every page in the weekly on-call handover: was it real, was it actionable, and did it fire early enough?
Label alerts with phone_number_id or tenant, so one failing number is not hidden inside a healthy global average.
Turn template status and phone-number quality webhooks into tickets — Meta often tells you about trouble before customers do.
Test alert rules like code: replay a recorded incident's metrics and check the rules would have fired, and when.

10Alerting FAQs

What burn rate should trigger a page?+

For a 30-day SLO, 14.4× over one hour is a common paging threshold, because it spends 2% of the month's budget in that hour. Pair it with 6× over six hours for slower burns and a 1× three-day ticket, and adjust if your window or traffic shape differs.

What is the difference between alerting and monitoring?+

Monitoring collects and displays data; alerting decides when that data needs a human. Dashboards and wider observability tooling help you diagnose once you are looking, while alerts are the small set of conditions worth interrupting someone for, ideally tied to an SLO.

How do I reduce alert fatigue?+

Delete or downgrade every alert that fires without requiring action. Move cause-based alerts to dashboards, group related alerts into one notification, add minimum-traffic conditions and route slow problems to tickets instead of the pager.

Should I alert on individual WhatsApp error codes?+

Usually not — alert on the overall send error ratio and break it down by error code on the dashboard. The exceptions are account-level signals such as sustained 130429 throttling, a template moving to PAUSED or a quality rating drop, which deserve their own ticket.

See InfiQ pricing

Alerting

How does alerting work?

Start from the SLO, not the dashboard

Compute the burn rate

Combine windows and thresholds

Add the alerts an SLO misses

Route, group and attach a runbook

02Try it

Where is the needle: page, ticket or sleep?

03What is alerting?

04Example: A bad deploy at a Hyderabad diagnostics chain

Alerting in code

Alerting: key facts and limits

Why does alerting matter?

Common mistakes with alerting

Alerting on causes instead of symptoms

Ratio alerts on tiny traffic

No alert for silence

Every alert is a page

Alerts without a runbook

Best practices for alerting

10Alerting FAQs

See how this works on InfiQ

From definition to your first message

See how this works on InfiQ