moby / moby/buildkit

handling of SIGTERM

Open
#3,282 0 comments 8 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
10.3k
Forks
1.5k
Avg merge
1d 23h
Merged PRs (30d)
48

Description

Currently buildkitd doesn't exit gracefully upon recieving SIGTERM.

$ docker run --detach --name buildkitd --privileged moby/buildkit:v0.10.5 
b982ae4d54669716e6c519b2a4044aa06ca398a0944c8c11502706273a1d4ee4
$ docker ps --filter name=buildkitd --all
CONTAINER ID   IMAGE                   COMMAND       CREATED          STATUS          PORTS     NAMES
b982ae4d5466   moby/buildkit:v0.10.5   "buildkitd"   42 seconds ago   Up 41 seconds             buildkitd
$ docker exec -ti buildkitd killall -TERM buildkitd
$ docker ps --filter name=buildkitd --all          
CONTAINER ID   IMAGE                   COMMAND       CREATED              STATUS                     PORTS     NAMES
b982ae4d5466   moby/buildkit:v0.10.5   "buildkitd"   About a minute ago   Exited (1) 7 seconds ago             buildkitd
$

A well-behaved process must handle SIGTERM by completing outstanding work items (as needed) and indicating that it has done so by returning 0.

Currently buildkitd logs look like this upon termination:

$ docker logs buildkitd
time="2022-11-14T11:03:00Z" level=info msg="auto snapshotter: using overlayfs"
time="2022-11-14T11:03:00Z" level=warning msg="using host network as the default"
time="2022-11-14T11:03:00Z" level=info msg="found worker \"nb4j68vntx10m5gi2qj5hbv2w\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:b982ae4d5466 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/arm64 linux/amd64 linux/amd64/v2 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/mips64le linux/mips64 linux/arm/v7 linux/arm/v6]"
time="2022-11-14T11:03:00Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2022-11-14T11:03:00Z" level=info msg="found 1 workers, default=\"nb4j68vntx10m5gi2qj5hbv2w\""
time="2022-11-14T11:03:00Z" level=warning msg="currently, only the default worker can be used."
time="2022-11-14T11:03:00Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
time="2022-11-14T11:04:09Z" level=info msg="stopping server"
buildkitd: context canceled
$

It would be more helpful if it also printed signal it received, and probably shouldn't print "context canceled", as it's meaningless to the user and doesn't adhere to the log format either.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the buildkitd entry point and reproduce termination with the Docker command shown in the issue. Trace the SIGTERM shutdown path and observe the exit status and logs. Done means outstanding work is handled as appropriate, the process exits with status 0, and termination output identifies the signal without the unhelpful "context canceled" message.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.