zombie containerd-shim processes

$ docker pull docker:20-dind
20-dind: Pulling from library/docker
Digest: sha256:4e1e22f471afc7ed5e024127396f56db392c1b6fc81fc0c05c0e072fb51909fe
Status: Image is up to date for docker:20-dind
docker.io/library/docker:20-dind

$ docker run -dit --privileged --name test docker:20-dind dockerd
1ee25dc98bf4bc5e232abe27a9e651b18cbfb8b3f6ca981c3ae64c894584e7b4
$ docker exec test ps faux
PID   USER     TIME  COMMAND
    1 root      0:00 dockerd
   33 root      0:00 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
  154 root      0:00 ps faux
$ docker exec test docker run --rm tianon/true
Unable to find image 'tianon/true:latest' locally
latest: Pulling from tianon/true
c53fb220cbad: Pulling fs layer
c53fb220cbad: Verifying Checksum
c53fb220cbad: Download complete
c53fb220cbad: Pull complete
Digest: sha256:009cce421096698832595ce039aa13fa44327d96beedb84282a69d3dbcf5a81b
Status: Downloaded newer image for tianon/true:latest
$ docker exec test ps faux
PID   USER     TIME  COMMAND
    1 root      0:00 dockerd
   33 root      0:00 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
  220 root      0:00 [containerd-shim]
  294 root      0:00 ps faux
$ docker exec test docker run --rm tianon/true
$ docker exec test docker run --rm tianon/true
$ docker exec test docker run --rm tianon/true
$ docker exec test ps faux
PID   USER     TIME  COMMAND
    1 root      0:00 dockerd
   33 root      0:00 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
  220 root      0:00 [containerd-shim]
  331 root      0:00 [containerd-shim]
  429 root      0:00 [containerd-shim]
  529 root      0:00 [containerd-shim]
  600 root      0:00 ps faux

If I do the same test with --init or ... docker:20-dind docker-init dockerd, then we get no zombies.

I think this is technically a bug in containerd, because I can reproduce with bare containerd as pid1 as well, but it doesn't seem quite the same as https://github.com/containerd/containerd/issues/5708 (although perhaps related).

cc @thaJeztah @cpuguy83

$ docker run -dit --privileged --name test --volume /var/lib/containerd docker:20-dind containerd
2fa1f7a0b543808572a7a2da7ad28fd165d783f1ac8f3e9c59ebb30417f43b9f
$ docker exec test ps faux
PID   USER     TIME  COMMAND
    1 root      0:00 containerd
   44 root      0:00 ps faux
$ docker exec test ctr i pull docker.io/tianon/true:latest
...
$ docker exec test ctr run --rm docker.io/tianon/true:latest foo
$ docker exec test ps faux
PID   USER     TIME  COMMAND
    1 root      0:00 containerd
  110 root      0:00 [containerd-shim]
  152 root      0:00 ps faux

Reproduce the commands in the issue using docker:20-dind, then compare normal execution with --init and with containerd as PID 1. Inspect the resulting containerd-shim processes and related containerd behavior, including the linked containerd issue, to identify why they persist. Done means repeated short-lived containers no longer leave zombie shims.