`docker system prune -af --volumes` does not completely clear out overlay2 folder

Description

We run a lot of docker containers in our CI/CD, the majority using the docker run --rm option if that matters. We also build a few images and perform a docker system prune -af --volumes each night. Every couple of months we end up with 0 bytes remaining disk space due to /var/lib/docker/overlay2 retaining data despite the nightly cleanup. The only way to recover the disk space without breaking docker that I could find so far is a complete reinstall.

For reference:

$ docker system prune --all --force --volumes
Total reclaimed space: 0B
$ docker ps --all
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
$ docker volume ls
DRIVER    VOLUME NAME

# du -ch /var/lib/docker --max-depth=1
4.0K    /var/lib/docker/runtimes
28K     /var/lib/docker/volumes
67G     /var/lib/docker/overlay2
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/swarm
11M     /var/lib/docker/buildkit
4.0K    /var/lib/docker/containers
106M    /var/lib/docker/image
16K     /var/lib/docker/plugins
68K     /var/lib/docker/network
67G     /var/lib/docker
67G     total

# df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   62G   46G   13G  79% /

# ls -lh /var/lib/docker/overlay2/
total 5.6M
drwx--x--- 5 root root 4.0K Oct 19 16:24 0042cff5c2b1153d4e2cc8d1249433c3fdf40a9af374de7372e0bb487fa59a0e
drwx--x--- 4 root root 4.0K Oct 19 16:24 0042cff5c2b1153d4e2cc8d1249433c3fdf40a9af374de7372e0bb487fa59a0e-init
drwx--x--- 4 root root 4.0K Oct 31 09:03 00656fdea360c00da5d9549adcb5f968ab19280e6dde08892f6d26901ba84cca
drwx--x--- 4 root root 4.0K Oct 30 12:47 0081118c886142094d4bfdac3051331b04a5937bc0cd9db5f4d292a793d54a33
drwx--x--- 4 root root 4.0K Oct 19 14:41 0124757a1161c7b9bc1f4a725a5ef90c7d0b9598d31e53c8c59780fc686c2ca4
...
drwx--x--- 4 root root 4.0K Nov  7 08:47 x32c6n0iual0i5x438yhm91zc
drwx--x--- 4 root root 4.0K Nov  6 08:49 x9g9q67ib6r4pr799rkjcacir
drwx--x--- 4 root root 4.0K Nov  9 07:45 yo8bov1h8hqzcpt3ubdsj3de1

# ls -lh /var/lib/docker/overlay2/ | wc -l
1388

# apt-get purge docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras
# rm -rf /var/lib/docker
# rm -rf /var/lib/containerd
# apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   62G   20G   40G  33% /

Reproduce

1. Intensive use of docker run --rm, docker rmi and docker build over weeks
2. Run docker system prune -af --volumes every day overnight
3. Observe disk filling up over time

Expected behavior

After running docker system prune -af --volumes, all data from dangling images, completed containers etc should be removed

docker version

**Note:** Below o/p is collected _after_ a reinstall. The version running for the past months has been 24.0.6.

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:01 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:01 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.15.0-76-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.58GiB
 Name: ****
 ID: 2c71d842-867e-40fa-af3e-a6210ada7edd
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

Start by reproducing the reported sequence with docker run --rm, docker rmi, docker build, and docker system prune -af --volumes, then inspect the CLI entry point for system prune and the /var/lib/docker/overlay2 state. The payload names no source file or test; done means identifying why reclaimable overlay2 data remains and adding coverage that verifies cleanup.