OpenTelemetry (OTel): Tracing Queues and Webhooks

Drawing 1 · Interactive

Follow one trace across the queue and back from Meta

4 / 4

javascript

// tracing.js — load first: node --require ./tracing.js app.js

const { NodeSDK } = require("@opentelemetry/sdk-node");

const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");

const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-http");

const sdk = new NodeSDK({

serviceName: process.env.OTEL_SERVICE_NAME || "whatsapp-send-worker",

traceExporter: new OTLPTraceExporter({

url: "http://otel-collector:4318/v1/traces",

}),

instrumentations: [getNodeAutoInstrumentations()],

});

sdk.start();

process.on("SIGTERM", () => sdk.shutdown().finally(() => process.exit(0)));

Step 4 of 4 · lines 14–15

Start the SDK, and flush buffered spans on shutdown so the last traces before a deploy are not lost.

Jump to

Three files that keep one OTP send in a single trace: SDK setup, context carried through the job queue, and a status-webhook span linked back to the send. Step through the highlighted lines.

What is OpenTelemetry?

OpenTelemetry (OTel) is an open-source observability framework, hosted by the Cloud Native Computing Foundation (CNCF), that standardises how software produces telemetry. It provides language APIs and SDKs for creating traces, metrics and logs; automatic instrumentation for common libraries such as HTTP servers and clients, database drivers and queue clients; semantic conventions that name attributes consistently; and OTLP, a wire protocol for shipping the data to any backend that accepts it. It was formed in 2019 by merging two earlier projects, OpenTracing and OpenCensus.

Its central idea is context propagation. Every trace has a 128-bit trace ID and every span a 64-bit span ID. When one service calls another over HTTP, the SDK injects them into the W3C traceparent header and the receiving service continues the same trace. A Message Queue has no HTTP call to ride on, so you inject the context into the job when you queue it and extract it in the worker. That is what turns an API request, a queue hop and a Graph API call into one connected trace.

Meta's webhooks are the hard edge. A status webhook arrives later, from Meta, with no traceparent of yours. The usual answer is to correlate by wamid, or to put your trace ID in biz_opaque_callback_data when sending, and then start a new span in the webhook handler that links back to the send. OpenTelemetry is the instrumentation standard; Distributed Tracing is the practice it enables, Structured Logging and the Four Golden Signals are fed by the same data, and Observability is the goal.

Key takeaways

OpenTelemetry is a CNCF open-source standard for producing traces, metrics and logs and shipping them over OTLP to any backend.
Context propagation is the core idea: the W3C traceparent header carries a 128-bit trace ID and a 64-bit span ID between services.
Across a job queue, inject the context into the job when queuing and extract it in the worker, or the trace breaks.
Meta's webhooks do not carry your trace context; link them back through the wamid or biz_opaque_callback_data.
Run a Collector to batch, sample and scrub personal data before telemetry leaves your network.

Implementation

OpenTelemetry in code

1# otel-collector.yaml
2receivers:
3  otlp:
4    protocols:
5      grpc:
6        endpoint: 0.0.0.0:4317
7      http:
8        endpoint: 0.0.0.0:4318
9 
10processors:
11  memory_limiter:
12    check_interval: 1s
13    limit_percentage: 80
14    spike_limit_percentage: 25
15  attributes/scrub:              # never export raw phone numbers or tokens
16    actions:
17      - key: whatsapp.recipient
18        action: delete
19      - key: http.request.header.authorization
20        action: delete
21  batch: {}
22 
23exporters:
24  otlphttp:
25    endpoint: https://otel-backend.example.com
26 
27service:
28  pipelines:
29    traces:
30      receivers: [otlp]
31      processors: [memory_limiter, attributes/scrub, batch]
32      exporters: [otlphttp]
33    metrics:
34      receivers: [otlp]
35      processors: [memory_limiter, batch]
36      exporters: [otlphttp]
37    logs:
38      receivers: [otlp]
39      processors: [memory_limiter, attributes/scrub, batch]
40      exporters: [otlphttp]

One Collector for all three signals: limit memory, delete sensitive attributes, batch, then export over OTLP.

Design intent

Why does OpenTelemetry matter?

Without a standard, every library, vendor agent and team invents its own instrumentation, and a trace stops at the first component that speaks a different dialect, which is usually the queue. OpenTelemetry means you instrument once and can change backends by changing an exporter rather than your code. In a WhatsApp pipeline it lets an engineer open one trace and see that a slow OTP spent 40 ms in your API, 11 seconds waiting in a queue and 300 ms in the Graph API call, instead of arguing about whose system is slow.

Operating sequence

How does OpenTelemetry work?

Start the SDK before your app
Load a tracing setup file first (for a CommonJS Node.js app, node --require ./tracing.js app.js) that sets the service name, an exporter and the auto-instrumentations.
Let auto-instrumentation cover the basics
HTTP servers and clients, database drivers and many queue clients are traced automatically, so incoming webhooks and outgoing Graph API calls get spans without code changes.
Propagate context through the queue
When queuing, propagation.inject() writes traceparent into the job; the worker calls propagation.extract() and starts its span inside that context, so one trace survives the asynchronous hop.
Add WhatsApp-specific attributes
Record wamid, phone_number_id, template name and Meta error codes as span attributes, and set the span status to error on failures, so traces are searchable by the identifiers support uses.
Ship through a Collector
Send OTLP to an OpenTelemetry Collector (port 4317 for gRPC, 4318 for HTTP) that batches, samples and scrubs personal data before exporting to the backend you choose.

Failure modes

Common mistakes with OpenTelemetry

Losing the trace at the queue
Auto-instrumentation rarely bridges a job queue on its own, so the worker starts a fresh, disconnected trace. Inject the context into the job and extract it in the worker.
Expecting Meta to propagate your context
Status webhooks come from Meta without your traceparent. Correlate by wamid, or carry your trace ID in biz_opaque_callback_data and use span links.
Phone numbers in attributes
Instrumentation can capture URLs, headers and bodies that contain personal data. Never set raw numbers as attributes, and delete sensitive keys in the Collector as a safety net.
Starting the SDK too late
If the app loads http or the database driver before the SDK starts, those libraries are never patched and spans go missing. Load the tracing file first with --require.
Keeping every trace
Tracing every webhook at broadcast scale is expensive. Use a parent-based ratio sampler, and tail sampling in the Collector to keep all traces that contain errors.

Specifications

OpenTelemetry: key facts and limits

Signals: Traces, metrics and logs
Wire protocol: OTLP over gRPC (port 4317) or HTTP (port 4318)
Propagation header: W3C traceparent: version-traceid-spanid-flags
Trace ID / span ID: 128-bit / 64-bit, hex-encoded
Origins: CNCF project, formed in 2019 from OpenTracing and OpenCensus
Sampling settings: OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG

Example: One instrumentation, two backends

A Delhi travel-booking platform ran a vendor tracing agent on its API and nothing on its RabbitMQ send workers, so every trace stopped at the queue and support could not follow an OTP past the first hop. The team replaced the agent with the OpenTelemetry Node.js SDK, injected traceparent into each job's headers and extracted it in the workers, and pointed everything at a Collector. Within a week, traces ran from the booking API to the Graph API call and on to the linked status webhook. The Collector's scrub processor also caught phone numbers that auto-instrumentation had picked up from URL query strings. Six months later the company changed tracing vendors by editing one exporter block in the Collector config, without touching application code.

Recommended practice

Best practices for OpenTelemetry

Name spans after operations (whatsapp.send, whatsapp.status), not URLs with IDs in them.
Write the trace ID into every structured log line so logs and traces join up.
Run a Collector even in small setups: it is where sampling, batching and scrubbing live.
biz_opaque_callback_data holds one string of up to 512 characters; if it also carries an idempotency key, store a single lookup ID there and keep the trace IDs in your send store.
Record Graph API error codes and the FB Trace ID (fbtrace_id) as span attributes on failures, so Meta support can find the same request.

OTel FAQs

What is the difference between OpenTelemetry and distributed tracing?+

Distributed tracing is the practice of following a request across services; OpenTelemetry is a standard toolkit for producing that data. OTel also covers metrics and logs, and it works with many tracing backends.

Is OpenTelemetry a monitoring tool?+

No. It produces and ships telemetry but does not store or display it. You send OTLP data to a backend of your choice, open-source or commercial, for storage, dashboards and alerts.

How do I trace a WhatsApp message through a job queue?+

Inject the trace context into the job when you queue it and extract it in the worker. Call propagation.inject() with the job as the carrier, then propagation.extract() and context.with() in the worker so its spans join the same trace.

Can OpenTelemetry trace Meta's webhooks back to my send?+

Not automatically, because Meta does not forward your trace headers. Store the wamid on the send span, or put your trace ID in biz_opaque_callback_data, and link the webhook span to the original trace.

See InfiQ pricing

OpenTelemetry(OTel)

Follow one trace across the queue and back from Meta

What is OpenTelemetry?

OpenTelemetry in code

Why does OpenTelemetry matter?

How does OpenTelemetry work?

Start the SDK before your app

Let auto-instrumentation cover the basics

Propagate context through the queue

Add WhatsApp-specific attributes

Ship through a Collector

Common mistakes with OpenTelemetry

Losing the trace at the queue

Expecting Meta to propagate your context

Phone numbers in attributes

Starting the SDK too late

Keeping every trace

OpenTelemetry: key facts and limits

Example: One instrumentation, two backends

Best practices for OpenTelemetry

OTel FAQs

See how this works on InfiQ

Put OTel to work

See how this works on InfiQ