Skip to content
Muhammet Şafak
tr
Asked by: Tuğçe Answered:

My Go worker in a container never receives SIGTERM and won't shut down gracefully—is this a PID 1 problem?


Question

I run a queue worker written in Go inside a Docker container, deployed on Kubernetes. In the code I have graceful-shutdown logic that listens for `os.Interrupt` and cancels a context when the channel fires. But on every deploy the worker does nothing during the grace period and eventually gets SIGKILLed. Looking at the logs, I noticed it never sees SIGTERM at all—as if the signal never reaches the worker. Is this a PID 1 problem, or am I missing something in my code?

Answer

Short answer: most likely yes, this is a PID 1 problem.

Short answer

But two separate bugs are in play at once: the signal probably never reaches the worker because of shell-form CMD, and your code listens for os.Interrupt (SIGINT) while missing SIGTERM.

Why

  1. Shell form makes you a child of /bin/sh -c. CMD ./worker (shell form) runs the worker under sh; sh as PID 1 does not forward SIGTERM to its child. That’s the number-one reason the signal never reaches the worker. The fix is exec form: CMD ["./worker"] or ENTRYPOINT ["/app/worker"].

    # forwards SIGTERM straight to the binary
    ENTRYPOINT ["/app/worker"]
  2. PID 1 has no default signal behavior. Even as PID 1 directly, the kernel does not apply default signal dispositions for PID 1; you must install the handler explicitly. If you only listen for os.Interrupt, you miss SIGTERM — and the signal Kubernetes sends on deploy is precisely SIGTERM.

    ctx, stop := signal.NotifyContext(context.Background(),
        syscall.SIGTERM, syscall.SIGINT)
    defer stop()
    // pass ctx into the worker loop; on cancel, stop pulling new jobs,
    // finish in-flight work, and exit.
  3. An entrypoint script breaks it too. If you use a shell entrypoint, the last line must be exec "$@". Run "$@" without exec and the shell stays PID 1 and swallows the signal again.

What to do

  1. Wire the signal → context cancel → drain chain. On SIGTERM, cancel the worker context, stop pulling new jobs, finish in-flight work, and exit. This must complete within terminationGracePeriodSeconds (default 30s); exceed it and you get SIGKILL. If you have long jobs, tune the grace period accordingly.

  2. You need zombie reaping if you spawn children. If the worker forks other processes, as PID 1 you must reap them; that means adding tini to the image yourself (Docker provides this via docker run --init, but Kubernetes has no built-in equivalent flag). For a single Go binary, exec form + correct signal handling is usually enough — no tini needed.

  3. Confirm the diagnosis, don’t guess. Checking whether PID 1 inside the container is really your binary or sh is a five-second job: docker exec <container> ps -o pid,comm. If you see sh at PID 1, the diagnosis is certain. Also make sure the image expects SIGTERM; STOPSIGNAL SIGTERM is the default, but if it’s set to something else your worker may be getting a signal it never listens for.

  4. On Kubernetes, cut traffic first. Graceful shutdown isn’t just catching a signal. When the pod is terminating, flip the readiness probe down and add a preStop hook (a short sleep) so the pod is removed from Service endpoints before SIGTERM arrives. Otherwise you can keep accepting new requests/jobs even while draining.

Bottom line: personally I’d first confirm PID 1 with docker exec ... ps, flip the Dockerfile to exec form, replace os.Interrupt with signal.NotifyContext(..., syscall.SIGTERM, syscall.SIGINT), and verify with one deploy that it drains within the grace period. I’d only add tini if the worker actually forks child processes; otherwise it’s a needless layer. The order is: first let the signal arrive, then let the code catch it, and last cut traffic in time.

Tags: #Docker#Go
Expertise: Backend Developer
Share:

Comments

Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.

More Questions

All questions

Search the site

Start typing to search posts, projects and pages.

Esc to close Powered by Pagefind