My Go worker in a container never receives SIGTERM and won't shut down gracefully—is this a PID 1 problem?
Question
I run a queue worker written in Go inside a Docker container, deployed on Kubernetes. In the code I have graceful-shutdown logic that listens for `os.Interrupt` and cancels a context when the channel fires. But on every deploy the worker does nothing during the grace period and eventually gets SIGKILLed. Looking at the logs, I noticed it never sees SIGTERM at all—as if the signal never reaches the worker. Is this a PID 1 problem, or am I missing something in my code?
Answer
Short answer: most likely yes, this is a PID 1 problem.
Short answer
But two separate bugs are in play at once: the signal probably never reaches the worker because of shell-form CMD, and your code listens for os.Interrupt (SIGINT) while missing SIGTERM.
Why
-
Shell form makes you a child of /bin/sh -c.
CMD ./worker(shell form) runs the worker undersh;shas PID 1 does not forward SIGTERM to its child. That’s the number-one reason the signal never reaches the worker. The fix is exec form:CMD ["./worker"]orENTRYPOINT ["/app/worker"].# forwards SIGTERM straight to the binary ENTRYPOINT ["/app/worker"] -
PID 1 has no default signal behavior. Even as PID 1 directly, the kernel does not apply default signal dispositions for PID 1; you must install the handler explicitly. If you only listen for
os.Interrupt, you miss SIGTERM — and the signal Kubernetes sends on deploy is precisely SIGTERM.ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT) defer stop() // pass ctx into the worker loop; on cancel, stop pulling new jobs, // finish in-flight work, and exit. -
An entrypoint script breaks it too. If you use a shell entrypoint, the last line must be
exec "$@". Run"$@"withoutexecand the shell stays PID 1 and swallows the signal again.
What to do
-
Wire the signal → context cancel → drain chain. On SIGTERM, cancel the worker context, stop pulling new jobs, finish in-flight work, and exit. This must complete within
terminationGracePeriodSeconds(default 30s); exceed it and you get SIGKILL. If you have long jobs, tune the grace period accordingly. -
You need zombie reaping if you spawn children. If the worker forks other processes, as PID 1 you must reap them; that means adding
tinito the image yourself (Docker provides this viadocker run --init, but Kubernetes has no built-in equivalent flag). For a single Go binary, exec form + correct signal handling is usually enough — no tini needed. -
Confirm the diagnosis, don’t guess. Checking whether PID 1 inside the container is really your binary or
shis a five-second job:docker exec <container> ps -o pid,comm. If you seeshat PID 1, the diagnosis is certain. Also make sure the image expects SIGTERM;STOPSIGNAL SIGTERMis the default, but if it’s set to something else your worker may be getting a signal it never listens for. -
On Kubernetes, cut traffic first. Graceful shutdown isn’t just catching a signal. When the pod is terminating, flip the readiness probe down and add a
preStophook (a short sleep) so the pod is removed from Service endpoints before SIGTERM arrives. Otherwise you can keep accepting new requests/jobs even while draining.
Bottom line: personally I’d first confirm PID 1 with docker exec ... ps, flip the Dockerfile to exec form, replace os.Interrupt with signal.NotifyContext(..., syscall.SIGTERM, syscall.SIGINT), and verify with one deploy that it drains within the grace period. I’d only add tini if the worker actually forks child processes; otherwise it’s a needless layer. The order is: first let the signal arrive, then let the code catch it, and last cut traffic in time.
Comments
Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.