Skip to content
Muhammet Şafak
tr
Asked by: Cem Answered:

How should I carry a correlation ID from the HTTP request all the way into queued jobs?


Question

I have a distributed system: a request hits my Laravel API, a job gets dispatched to SQS there, and then a worker I wrote in Go picks up the message and processes it. The problem is I can't trace a request end to end — I have no common thread to tie an error in the Laravel logs to an error in the Go worker. I want to generate a correlation ID, but I'm not sure where to generate it, how to carry it into SQS, and how to read it back in the Go worker and attach it to every log line. Should I put it in the message body? Or invent my own header?

Answer

Short answer: put the correlation ID in the transport metadata (an SQS message attribute), not the business body, and inject it into context at every boundary it crosses. Don’t invent your own header — use W3C traceparent so you get end-to-end OpenTelemetry tracing for free later.

  1. Capture the source in one place. In Laravel, an inbound middleware should check whether the incoming request carries a traceparent; if not, generate one. Put that value into both the request context and the logger’s persistent context — now every log line in that request carries the ID.
  2. Put it on the attribute, not the payload. The ID isn’t business data; write it into SQS MessageAttributes when you dispatch the job. That keeps your serialized payload schema clean, and the worker can read the ID without parsing the message body.
  3. Re-hydrate it in the Go worker. When you receive the message, load the attribute into a carrier, extract it into the context, then seed the logger with that ID:
carrier := propagation.MapCarrier{}
for k, v := range msg.MessageAttributes {
    carrier[k] = aws.ToString(v.StringValue)
}
ctx = otel.GetTextMapPropagator().Extract(ctx, carrier)
cid := trace.SpanContextFromContext(ctx).TraceID().String()
logger := slog.With("trace_id", cid)
  1. Pick a standard: W3C Trace Context. traceparent is parsed by ready-made libraries in both languages; if you invent your own format you’ll have to re-implement it in every service and you cut yourself off from the OTel ecosystem.
  2. Align the log format. Use structured JSON logs on both sides and write the same field name (trace_id). If the field names disagree, you can’t join the two services with a single query in OpenSearch/CloudWatch.
  3. Preserve it through retries and DLQ. A message attribute travels with the message; even if the job is retried or lands in the DLQ, the ID survives — which keeps the thread intact exactly when you need it most (during a failure).

Bottom line: personally I’d generate the ID as a W3C traceparent from the start, carry it in an SQS message attribute, inject/extract it with the OTel propagator on both sides, and feed it into the structured logger as trace_id. That gives you a greppable correlation ID today and leaves no extra work to graduate to real distributed tracing along the same line.

Tags: #observability#logging#queues
Share:

Comments

Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.

More Questions

All questions

Search the site

Start typing to search posts, projects and pages.

Esc to close Powered by Pagefind