How do I set up distributed tracing (OpenTelemetry) across microservices?
Question
A request travels API Gateway → Auth → Order → Stock. We're struggling to find the source of an 800ms delay in one of the services in between. With OpenTelemetry we plan to carry a `trace_id` across all services (Laravel and Go) and visualize it in a central APM (Jaeger/Grafana Tempo). How do I design context propagation over HTTP headers and queue messages?
Answer
Short answer: don’t invent a custom scheme; standardize on W3C Trace Context (traceparent/tracestate) so PHP and Go interoperate without friction.
Your real problem is “where’s the 800ms” — and to see it the trace has to travel across services and queues without breaking. Here’s the mechanism:
- Make W3C Trace Context the standard. Start the root span at the gateway; every service extracts context from the incoming HTTP header and injects it into outgoing calls. Because it’s a standard header, Laravel and Go read the same
traceparentin common — don’t invent your owntrace_idheader. - Carry the same
traceparentthrough queues. When you publish a message, addtraceparentas a message header/attribute and re-extract it in the worker. This is exactly the piece teams forget; skip it and async hops show up as orphan traces in the APM and the chain breaks. - Put the OTel SDK in each service, ship via OTLP. Service → OTLP → a Collector → Jaeger/Tempo. A Collector in between decouples export from the app and makes swapping backends easy.
- Choose sampling smartly. Use
parent-basedso all spans of a trace share the same fate; add tail sampling on top so you definitely keep the 800ms outliers. With head-only sampling you drop the very slow trace you wanted to see. - Don’t hand-roll trace_id. Don’t write your own header read/write code; the SDK’s context propagation already carries it. Do it by hand and you’ll forget it somewhere and the trace snaps there.
Bottom line: I’d set up W3C Trace Context + the OTel SDK + a Collector + Tempo/Jaeger, leave propagation to the SDK, and never skip putting traceparent into the message header on the queue hop. I explain the architectural reasoning for why I build it this way on sade.dev; you’ll find the detail of the Go-side HTTP mechanics in the hub post. Only an unbroken trace shows you the 800ms.
Related Reading
Comments
Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.