Interrupted container removal can leave a Dead:false container dir that survives startup cleanup and nondeterministically shadows its replacement's name on every daemon restart

Summary

Container removal (daemon/delete.go cleanupContainer) is non-atomic:

1. stop container
2. ctr.State.Dead = true; CheckpointTo(disk)     // write failure is logged and SWALLOWED (delete.go:135)
3. imageService.ReleaseLayer(rwLayer)            // deletes layerdb mount record + graphdriver storage
4. containerfs.EnsureRemoveAll(ctr.Root)         // deletes /var/lib/docker/containers/<id>/
5. deregister name / delete from container map

If the daemon dies (or the step fails) between 3 and 4, the container's directory —
config.v2.json included — survives on disk while its layer is gone. On the next daemon
start, restore() loads the leftover dir, GetLayerByID fails
(failed to load container mount ... mount does not exist), and — since #51724 — the
container is registered anyway with a nil RWLayer.

The startup cleanup added in #51692 removes such leftovers only when State.Dead is true
on disk. Two paths produce leftovers where it is false, and those survive forever:

1. Swallowed checkpoint failure: the Dead=true CheckpointTo at step 2 can fail
   (ENOSPC, I/O error); the error is logged ("Error saving dying container to disk") and
   removal continues. If a later step then fails or the daemon dies, the on-disk config
   still has Dead:false.
2. Crash consistency: the checkpoint is an atomic-rename write, the layer release is a
   set of separate filesystem operations. A host crash (as opposed to a daemon kill) can
   persist the layer release but not the config rename. After reboot the surviving config is
   the pre-removal one — Dead:false.

Because the replacement container (created under the same name after the faulty removal) and
the leftover dir now both claim one name, every daemon restart re-races them:
restore() loads containers from a directory scan in enumeration order, first
registerName wins, the loser is dropped from the container map — invisible to
docker ps -a, unremovable via the API, and untouched on disk. Which copy wins is
nondeterministic and genuinely swaps between restarts (observed repeatedly on one host).
When the broken copy wins, the user sees:

Error response from daemon: RWLayer of container <id> is unexpectedly nil

on start/inspect of a container that worked fine the previous boot — i.e. "containers
randomly break after reboot, a different set each time."

Environment

• First observed on a host where a since-removed third-party Unraid plugin
  (RAM-DISK-Dockerlog) kept /var/lib/docker/containers on tmpfs with a badly-stale
  (~6h) sync back to real storage, so unclean shutdowns interrupted removals far more often
  than stock config would. That plugin is fully uninstalled and is not the mechanism being
  reported here — it was only what made the underlying non-atomic-removal defect frequent
  enough to characterize. The exact same signature recurred on this host on multiple reboots
  after its complete removal, confirming the defect is in Docker Engine itself.
• Docker Engine 29.5.2 (also reproduced against 29.5.3), containerd-snapshotter=false
• Storage driver: btrfs (mechanism is storage-driver-agnostic; only the restore-time error
  text is driver-specific)
• Unraid 7.3.1 (Slackware-based), kernel 6.18.33
• Long-lived real-world specimens observed with exactly this signature:
  containers/<id>/ present with Dead:false, RemovalInProgress:null, stale state;
  image/btrfs/layerdb/mounts/<id>/ absent.

Reproduction (daemon-kill variant; produces the Dead:true flavor)

In an isolated dockerd (we used a privileged DinD container so the kill loop is harmless):

docker run -d --name victim busybox sleep 300
ID=$(docker inspect -f '{{.Id}}' victim)
# dilate the removal window (optional): mkdir + create ~40k files under
#   /var/lib/docker/containers/$ID/ so EnsureRemoveAll takes observable time
docker rm -f victim &
# poll for the instant the layer record disappears, then kill the daemon:
while [ -d /var/lib/docker/image/btrfs/layerdb/mounts/$ID ]; do :; done
kill -9 $(pidof dockerd)

Result (first attempt, in our runs): containers/$ID/ present with intact
config.v2.json, image/btrfs/layerdb/mounts/$ID/ gone. On daemon restart:
failed to load container mount ... mount does not exist, container registered with nil
RWLayer. This flavor has Dead:true and is cleaned by the #51692 pass at the next
restart — the report is about the two Dead:false paths above, which are not.

For the swallowed-checkpoint path, fault-inject a failure at CheckpointTo in step 2 (or
fill the filesystem); removal proceeds, and any subsequent interruption yields the
Dead:false survivor.

Secondary observation: DELETE reports success while a removal step fails

Live capture on the same host (29.5.2): force-removing a container whose graphdriver
storage was already missing returned HTTP 204 while the daemon logged

level=error msg="Error removing mounted layer <id>: stat /var/lib/docker/btrfs/subvolumes/<mount-id>: no such file or directory"

and left the layerdb mount record on disk (the mirror image of the main signature). A
caller cannot detect the partial failure even if it checks the response.

Suggested directions

• Extend the #51692 startup cleanup: a restored container whose rwlayer fails to load and
  whose name is already registered to another loadable container is unambiguously residue —
  remove (or quarantine) it instead of silently dropping it from the map.
• Alternatively, make registerName collisions prefer the container whose rwlayer loaded.
• Treat a failed Dead=true checkpoint as fatal for the removal (retry/error out) rather
  than proceeding to release the layer with the on-disk state still saying the container is
  alive.

Prior-art search

Tracker searched (2026-08-26) for "failed to load container mount", "name is reserved" +
"failed to register", "RWLayer" "unexpectedly nil", and issues referencing #51692/#51724:
no existing open issue covers the Dead:false leftover-directory case or the restart-time
name race with a same-name replacement. Closest matches are #51692/#51724 themselves (both
merged; cover only the Dead:true flavor and registration respectively) and #21135 (a 2016
v1.9→v1.10 migration-era "name is reserved", unrelated mechanism).

References

• #51692 (daemon: clean up dead containers on start) — covers only State.Dead == true
• #51724 (daemon: restore: register containers without rwlayer)
• Independent user reports of the visible symptom on Unraid 7.3.0:
  https://forums.unraid.net/topic/198745-unraid-730-stable-now-available/page/5/
  ("dockers disappeared and if you reboot different ones came and went")

Start with daemon/delete.go cleanupContainer, then trace restore() and registerName() using the leftover-directory scenario described in the issue. Reproduce the daemon-kill variant if possible and compare the Dead:true and Dead:false cases, including the failed CheckpointTo path. Done means interrupted removal cannot leave an unhandled stale container that races a same-name replacement after restart, and partial removal failures are not silently reported as success.