01 · Definition
What is distributed tracing?
Distributed tracing is the practice of following a single unit of work (an API call, an OTP send, an inbound customer message) as it moves through several services, and recording each step as a span: a named, timed operation with a start, a duration, a status and attributes. Every span in the journey shares one trace ID and points to its parent span, so a tracing backend can redraw the whole path as a waterfall and show exactly where time and errors accumulated.
A WhatsApp send is a good example of why it matters. Your API accepts the request and writes to a database, a message queue holds the job, a worker builds the template and calls the Cloud API, and minutes later Meta's statuses webhook reports sent, delivered or failed. Each piece logs on its own; only a trace ties them together, so you can see that a slow OTP spent a second waiting behind a broadcast in the queue rather than in Meta's API.
Traces need context propagation: each hop passes the trace ID and parent span ID to the next. Between HTTP services this travels in the W3C traceparent header; through a queue it travels in the job's headers or metadata. Meta does not propagate your context, and its webhooks arrive as fresh requests, so integrations store the WhatsApp Message ID (WAMID) returned by the send and use it to link the later status spans to the original trace.
Tracing complements rather than replaces the other parts of observability. Structured logging records what happened inside one step, metrics count how often, and traces show the causal chain across steps. OpenTelemetry (OTel) is the vendor-neutral standard most teams use to emit spans; distributed tracing is the practice it enables.
Key takeaways
- Distributed tracing records one request as a tree of timed spans that share a trace ID, so you see the whole path, not isolated log lines.
- Context must be carried across every hop: HTTP headers between services and message headers through queues.
- Meta's webhooks carry no trace header, so link status-webhook spans back to the original send by its wamid.
- Keep phone numbers and message text out of span attributes; record IDs such as
phone_number_idandwamidinstead. - Sample by keeping every error and slow trace, then a small share of normal ones; random head sampling throws rare failures away.
02 · Code samples
Distributed tracing in code
import { trace, context, propagation, SpanKind, SpanStatusCode } from "@opentelemetry/api";const tracer = trace.getTracer("wa-sender");// Producer: put the current trace context into the job's headers.export async function enqueueSend(job) {await tracer.startActiveSpan("wa-send publish", { kind: SpanKind.PRODUCER }, async (span) => {const headers = {};propagation.inject(context.active(), headers); // adds traceparentawait queue.add("wa-send", { ...job, otel: headers });span.end();});}// Worker: continue the SAME trace, then record what Meta returned.export async function handleSend(job) {const parent = propagation.extract(context.active(), job.data.otel);await context.with(parent, () =>tracer.startActiveSpan("graph messages send", { kind: SpanKind.CLIENT }, async (span) => {span.setAttribute("whatsapp.phone_number_id", job.data.phoneNumberId);const res = await fetch(`https://graph.facebook.com/v26.0/${job.data.phoneNumberId}/messages`, {method: "POST",headers: { Authorization: `Bearer ${process.env.ACCESS_TOKEN}`, "Content-Type": "application/json" },body: JSON.stringify(job.data.payload),});const body = await res.json();span.setAttribute("http.response.status_code", res.status);if (body.messages?.[0]?.id) {span.setAttribute("whatsapp.wamid", body.messages[0].id);await traceStore.save(body.messages[0].id, span.spanContext()); // for webhook links}if (body.error) {span.setAttribute("whatsapp.error_code", body.error.code);span.setAttribute("whatsapp.fbtrace_id", body.error.fbtrace_id);span.setStatus({ code: SpanStatusCode.ERROR, message: body.error.message });}span.end();}),);}
Inject on publish, extract in the worker, and store the span context against the wamid so later status webhooks can link back.
03Playground
A working model of Distributed Tracing. Use its controls to replay it step by step.
Where did this OTP spend its 3.4 seconds?
3,434 ms end to end · 8 spans
Bottleneck: Queue · waiting behind broadcast takes 1,150 ms, 33% of the trace. Speeding it up shortens the whole request the most; trimming a span that runs in parallel with it may save nothing.
Whole trace: 3,434 ms across 8 spans. Bottleneck: Queue · waiting behind broadcast, 1,150 ms (33% of the trace).
04 · Operation
How does distributed tracing work?
Start a root span at the edge
When your API receives the send request (or your webhook receiver gets an inbound message), start a span. Instrumentation libraries do this automatically for common HTTP frameworks.
Propagate context across every hop
Outgoing HTTP calls carry
traceparent. When you enqueue a job, inject the context into the job's headers; the worker extracts it and continues the same trace instead of starting a new one.Wrap the Graph API call in a client span
Record the HTTP status, the
phone_number_id, the returnedwamidand, on errors, Meta's errorcodeandfbtrace_idas attributes, and mark the span status as an error.Link status webhooks by wamid
Save the trace and span IDs against the wamid when the send succeeds. When a status webhook arrives, start a consumer span with a span link to that saved context, so delivery time joins the original story.
Sample, store and query
Export spans to a tracing backend. Keep every trace with an error or high latency (tail-based sampling) plus a small share of normal traffic, and search by wamid, trace ID or error code during incidents.
05 · Worked example: The OTP that waited behind a sale
Example: The OTP that waited behind a sale
A Pune fintech sees OTP complaints every evening at 7 pm. Dashboards show the Cloud API responding in about 400 ms and the queue "healthy". A trace for one slow OTP tells a different story: 48 ms in the API, then 1.15 seconds waiting in the shared send queue behind a marketing broadcast, then a normal Graph API call. The fix is not a faster server but a separate high-priority queue for authentication templates. The next evening's traces show queue waits under 50 ms, and the delivered-status span arrives roughly a second sooner.
06 · Specifications
Distributed tracing: key facts and limits
| Trace ID | 16 bytes, written as 32 hex characters (W3C Trace Context) |
|---|---|
| Span ID | 8 bytes, written as 16 hex characters |
| Propagation header | traceparent (plus optional tracestate) |
| Through a queue | Inject into job headers, extract in the worker |
| Meta webhooks | No trace header: link statuses by wamid |
| Meta-side request ID | fbtrace_id in Graph API error bodies |
| Sampling | Head-based (decided at start) or tail-based (decided after) |
07 · Cautions
Common mistakes with distributed tracing
Losing the trace at the queue
Auto-instrumentation covers HTTP but often not your job payloads, so every worker starts a new, orphaned trace. Inject the context when you enqueue and extract it in the worker; check one end-to-end trace after every queue or library change.
Expecting Meta to continue your trace
The Cloud API ignores your
traceparent, and webhooks start fresh. Store the span context against the wamid and use span links, and recordfbtrace_idon failed calls so Meta support can find the request on their side.Putting personal data in attributes
Phone numbers, names and message bodies in spans end up in a third-party backend with long retention. Record IDs (
wamid,phone_number_id, template name) and mask anything personal before export.Sampling away the failures
Keeping a random 1% of traces at the start of each request means the rare failed OTP is almost never kept. Use tail-based sampling that always keeps errors and slow traces.
High-cardinality span names
Naming spans after full URLs (
POST /v26.0/106540352242922/messages) explodes the number of distinct operations. Use a stable name such asgraph messages sendand put the IDs in attributes.
08 · Rationale
Why does distributed tracing matter?
Most WhatsApp latency complaints are not about one slow service but about the gap between services: a job waiting in a queue, a retry loop, a CRM call inside the webhook handler. Averages and per-service dashboards hide that. A trace answers "where did the four seconds go?" for a specific message in one view, which shortens incidents, settles arguments about whose system is slow, and shows which fix (a priority queue, a timeout, a cache) will actually move P99 latency.
09 · Recommended practice
Best practices for distributed tracing
- Put the trace ID in every structured log line so you can jump from a log to its trace and back.
- Add a span for time spent waiting in the queue, not just for processing; waiting is where broadcasts hurt OTPs.
- Return the trace ID in an
X-Request-Id-style response header to your own clients to speed up support tickets. - Trace the inbound path too: webhook receipt, publish, bot or inbox handling, and the reply send.
10 · Questions and answers
Distributed Tracing FAQs
- What is the difference between distributed tracing and logging?
- Logs record isolated events inside one service; a trace links the steps of one request across services. Logs tell you what a worker did, while the trace shows that the job waited 1.2 seconds in the queue before that worker picked it up. Most teams correlate them by writing the trace ID into every log line.
- Does the WhatsApp Cloud API support the traceparent header?
- No, Meta does not continue your trace or send trace headers on webhooks. Treat the Graph API call as a client span, record the returned wamid, and link later status webhooks to it; on errors, keep the
fbtrace_idMeta returns for support cases. - What is the difference between a trace ID and a wamid?
- A trace ID identifies one journey through your systems; a wamid identifies one WhatsApp message on Meta's side. One trace can contain the wamid as an attribute, and the wamid is the key that joins Meta's status webhooks back to your trace.
- Do I need OpenTelemetry to do distributed tracing?
- No, but it is the usual choice. OpenTelemetry gives vendor-neutral APIs, SDKs and context propagation, so you can switch tracing backends without re-instrumenting. Any tool works as long as every hop propagates the context.