End-to-End Observability: Distributed Tracing with OpenTelemetry and Prometheus

1. W3C Traceparent Header Propagation

A distributed trace is only as good as its propagation — every service in a request path needs to agree on how a trace's identity moves across network boundaries, or you get disconnected fragments instead of one coherent picture. The W3C Trace Context specification standardizes this as the `traceparent` header: a fixed-format string encoding a version, a 16-byte trace ID shared by every span in the request, an 8-byte parent span ID identifying the immediate caller, and trace flags indicating whether this trace is being sampled.

Before this standard existed, every vendor (Zipkin's B3 headers, Jaeger's own format, AWS X-Ray's format) used incompatible propagation headers, which meant a request crossing from a service instrumented with one vendor's SDK to a service instrumented with another's would silently break the trace into two unrelated ones. OpenTelemetry's SDKs implement W3C Trace Context by default, and virtually every observability backend (Datadog, Honeycomb, Grafana Tempo, AWS X-Ray via an adapter) now understands it, which is what makes OpenTelemetry a genuinely vendor-neutral instrumentation layer rather than another proprietary format to lock into.

In practice, propagation is mostly automatic once you instrument your HTTP client and server libraries correctly — the OpenTelemetry SDK's auto-instrumentation for `fetch`, `http`, and common frameworks injects and extracts the header transparently. The place it breaks is asynchronous boundaries the SDK can't see automatically: message queues, background job systems, and webhooks, where you must manually propagate the trace context by serializing it into the message payload and re-extracting it on the consumer side.

The `tracestate` header, which travels alongside `traceparent`, is worth understanding even though it rarely needs manual handling: it carries vendor-specific key-value pairs (a Datadog-specific priority sampling decision, for instance) that a multi-vendor pipeline needs to preserve across hops even when it can't interpret them. Dropping `tracestate` while forwarding `traceparent` is a subtle bug that silently breaks sampling decisions or vendor-specific enrichment downstream, without breaking the trace's basic connectivity — which makes it easy to miss until a specific vendor integration stops working correctly.

2. Structured Logging with Trace Correlation

Traces tell you where time went; logs tell you what actually happened at a specific point. They become dramatically more useful together than either alone, but only if every log line emitted during a request carries that request's trace and span ID, so you can pivot from a slow span in your tracing UI directly to the exact log lines emitted during that span, instead of grepping timestamps and hoping they line up.

This requires structured (JSON) logging rather than free-text logs, because you need machine-readable fields (`trace_id`, `span_id`) attached to every entry, and it requires wiring your logger to read the active OpenTelemetry context at log time — most logging libraries (Pino, Winston, structlog) have an OpenTelemetry integration or a small middleware that injects these fields automatically from the active span context, so application code doesn't need to thread trace IDs through manually.