Constantly growing /var/lib/docker/overlay2/

Description

We have a set of job-runners that perform various build- and deployment-related tasks using docker. After some time (weeks), all of them end up with critical disk usage and have to be re-provisioned as there does not appear to be a way to release the used disk space via the docker CLI:

$ du -sh /var/lib/docker/*
72.4M	/var/lib/docker/buildkit
71.2M	/var/lib/docker/containers
4.0K	/var/lib/docker/engine-id
293.9M	/var/lib/docker/image
132.0K	/var/lib/docker/network
65.7G	/var/lib/docker/overlay2
0	/var/lib/docker/plugins
0	/var/lib/docker/runtimes
0	/var/lib/docker/swarm
0	/var/lib/docker/tmp
5.0G	/var/lib/docker/volumes
$ ls -l /var/lib/docker/overlay2/ | wc -l
4204
$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          2         2         123.7MB   0B (0%)
Containers      2         2         168B      0B (0%)
Local Volumes   62        0         5.292GB   5.292GB (100%)
Build Cache     0         0         0B        0B
$ docker builder ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT             PLATFORMS
default * docker                                       
  default default         running v0.11.7+d3e6c1360f6e linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
$ docker builder du
Reclaimable:	0B
Total:		0B
$ docker system prune -a -f
Total reclaimed space: 0B
$ docker builder prune -a -f
Total:	0B

Related to #32420, #43586

Reproduce

I'm not sure as it appears to be building up slowly over time.

Expected behavior

The used disk space under /var/lib/docker/overlay2/ shows up in any diagnostic docker ... command and can be released with any (other) docker ... command.
I would expect it to show up under docker system df and/or docker builder du and for it t be releaseable with docker system prune or docker builder prune.

docker version

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:09:13 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:07:45 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.14.0-70.13.1.el9_0.x86_64
 Operating System: Rocky Linux 9.1 (Blue Onyx)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.28GiB
 Name: xxx
 ID: c603ff4f-a1fb-4a1c-b323-75ef6e588528
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: xxx
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

Start by comparing the reported /var/lib/docker/overlay2/ usage with docker system df and docker builder du, then test whether docker system prune or docker builder prune accounts for or releases it. Done means the accumulated overlay2 space is correctly represented by a diagnostic command and can be released through an appropriate Docker CLI command.