Volumes Returned by List Volumes Filtered Dangling API Return In-Use Containers

Description

This may be related to (and fixed by) #44512, but I wanted to bring it up as separate issue just in case.

We are gathering a list of dangling volumes, iterating over the list, deleting each one.

However, some of the dangling volumes (i.e., not referenced by any containers) are failing to delete with the error:

com.github.dockerjava.api.exception.ConflictException: Status 409: remove 74c0c8cb7c14b77eb445142018b31309ef8ecf0cbbeb7eff0874c2e0fc8f6170: volume is in use - [7dd18196c01196eb547ed31fee9d3cee4b9b644e11c33e713d04fe62f4031737]

Volumes that fail to delete in this manner are successfully deleted on a later retry.

As an example:

# Consider a88def775aa002b128e8e27d9a8f311bebfaa78404be73c7b3c288daa9167dbe

# Register the new volume
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.190146857Z" level=debug msg="Registering new volume reference: driver \"local\", name \"a88def775aa002b128e8e27d9a8f311bebfaa78404be73c7b3c288daa9167dbe\""

# Get the volumes (with dangling filter)
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.190859107Z" level=debug msg="Calling GET /v1.23/volumes?filters=%7B%22dangling%22%3A%5B%22true%22%5D%7D"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.190954815Z" level=debug msg=VolumeStore.Find ByType=service.andCombinator ByValue="[0x55f1071c3500 false]"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.191652378Z" level=debug msg="Probing all drivers for volume with name: 101f4c3de9ee7465f9b5aa6c86d681dd99362f301eccac86044b89a90b33a72c"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.191704707Z" level=debug msg="Registering new volume reference: driver \"local\", name \"101f4c3de9ee7465f9b5aa6c86d681dd99362f301eccac86044b89a90b33a72c\""
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.193239809Z" level=debug msg="Probing all drivers for volume with name: 45d1ac9b0bacf8f18e93674be97bda4d0642c382e4f2bc38b2a0c8bcd421e96f"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.193288732Z" level=debug msg="Registering new volume reference: driver \"local\", name \"45d1ac9b0bacf8f18e93674be97bda4d0642c382e4f2bc38b2a0c8bcd421e96f\""
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.195397265Z" level=debug msg="Probing all drivers for volume with name: 0341d0dd59bdd9a54228f6be4d5b82fdb689b2f9b90b91740c3698f50fc85809"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.195472524Z" level=debug msg="Registering new volume reference: driver \"local\", name \"0341d0dd59bdd9a54228f6be4d5b82fdb689b2f9b90b91740c3698f50fc85809\""
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.196706351Z" level=debug msg="Probing all drivers for volume with name: 75218b76b4d9462867ff86684751d54b85fe2c579d9c0ad7ade8a49b3fa2c9df"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.196786971Z" level=debug msg="Registering new volume reference: driver \"local\", name \"75218b76b4d9462867ff86684751d54b85fe2c579d9c0ad7ade8a49b3fa2c9df\""
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.198467615Z" level=debug msg="copying image data from 3f88f593a7763ee17ade04df993fd705bda0909777dac6d9f09f894061b78679:/data01, to fbc54eb92fa771703c8a03346dd947f1d0d19b6cdefa71033e204ae829d3c161"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.198530543Z" level=debug msg="copying image data from 3f88f593a7763ee17ade04df993fd705bda0909777dac6d9f09f894061b78679:/root, to 98cdbc6ef31800517bb7e9ab97a30311a195458a74b998914f7e73ecd746e539"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.198718058Z" level=debug msg="copying image data from 3f88f593a7763ee17ade04df993fd705bda0909777dac6d9f09f894061b78679:/run, to a999ab26361ae7a0df1c31ae71fd74bb82a0020aa2076514949f02ed0bc1659b"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.201148603Z" level=debug msg="copying image data from 3f88f593a7763ee17ade04df993fd705bda0909777dac6d9f09f894061b78679:/private, to 3434fdb44fe71dddd7f10d77cd83a1a4d257d98f2567d78c0d4622c01f5942ca"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.201209862Z" level=debug msg="copying image data from 3f88f593a7763ee17ade04df993fd705bda0909777dac6d9f09f894061b78679:/data02, to 589f367db89203d5be640e107722eb70aeccfd1217e39f0ffb3b459eb9fbca1c"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.201245366Z" level=debug msg="copying image data from 3f88f593a7763ee17ade04df993fd705bda0909777dac6d9f09f894061b78679:/opt, to 0341d0dd59bdd9a54228f6be4d5b82fdb689b2f9b90b91740c3698f50fc85809"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.203043374Z" level=debug msg="copying image data from b79f0c8d366ce63e83c2158f3e69f174488d08d8a465b3a015537edf9b75f6e7:/data01, to 096a286b24cd9231fbeb41448b3b2dc3a20428f872739474d59c1ae1b04be949"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.203104635Z" level=debug msg="copying image data from b79f0c8d366ce63e83c2158f3e69f174488d08d8a465b3a015537edf9b75f6e7:/root, to dddc83f37a9f91aa1598c61c38d21924c453a795f718de04d76342da8e8a5573"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.203280672Z" level=debug msg="copying image data from b79f0c8d366ce63e83c2158f3e69f174488d08d8a465b3a015537edf9b75f6e7:/private, to 8b306a1f019fafbd997e548d5393c3554384d74f8e8d43fcb7a22fd9971e9cdd"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.203321239Z" level=debug msg="copying image data from b79f0c8d366ce63e83c2158f3e69f174488d08d8a465b3a015537edf9b75f6e7:/opt, to a88def775aa002b128e8e27d9a8f311bebfaa78404be73c7b3c288daa9167dbe"
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.224679733Z" level=debug msg="Calling DELETE /v1.23/volumes/3434fdb44fe71dddd7f10d77cd83a1a4d257d98f2567d78c0d4622c01f5942ca"

# Attempting to delete the target volume because it was returned in the list of dangling volumes - fails
Dec 14 21:50:28 host.domain dockerd[20062]: time="2022-12-14T21:50:28.242236191Z" level=debug msg="Calling DELETE /v1.23/volumes/a88def775aa002b128e8e27d9a8f311bebfaa78404be73c7b3c288daa9167dbe"
Dec 14 21:50:29 host.domain dockerd[20062]: time="2022-12-14T21:50:29.777067140Z" level=debug msg="uploaded layer sha256:b930bd81aea92720aac658fe6b28a4453b714144eb75158961cd8085bff3f2d1 (sha256:d09e88c5d42bb5dd9ec754ae6a67a68216c5bcb5fc752a05afe468123f0e2e5b), 81196715 bytes"
Dec 14 21:50:29 host.domain dockerd[20062]: time="2022-12-14T21:50:29.917729542Z" level=debug msg="copying image data from b79f0c8d366ce63e83c2158f3e69f174488d08d8a465b3a015537edf9b75f6e7:/run, to b76eff516b97ce0a3b132f5b1a381805e75518a1fba9418ea5bb46f65a516689"
Dec 14 21:50:29 host.domain dockerd[20062]: time="2022-12-14T21:50:29.920051731Z" level=debug msg="copying image data from b79f0c8d366ce63e83c2158f3e69f174488d08d8a465b3a015537edf9b75f6e7:/var, to fd0994af70c8ca6eafbd2244f19a96ed5a2672a751a8d08302e7708efeedae29"
Dec 14 21:50:29 host.domain dockerd[20062]: time="2022-12-14T21:50:29.946944939Z" level=debug msg="copying image data from 3f88f593a7763ee17ade04df993fd705bda0909777dac6d9f09f894061b78679:/var, to 101f4c3de9ee7465f9b5aa6c86d681dd99362f301eccac86044b89a90b33a72c"
Dec 14 21:50:30 host.domain dockerd[20062]: time="2022-12-14T21:50:30.485558352Z" level=debug msg="copying image data from 3f88f593a7763ee17ade04df993fd705bda0909777dac6d9f09f894061b78679:/tmp, to 45d1ac9b0bacf8f18e93674be97bda4d0642c382e4f2bc38b2a0c8bcd421e96f"
Dec 14 21:50:30 host.domain dockerd[20062]: time="2022-12-14T21:50:30.512319270Z" level=debug msg="copying image data from b79f0c8d366ce63e83c2158f3e69f174488d08d8a465b3a015537edf9b75f6e7:/tmp, to d412c163b365a2d1e1ab712055e702a9b9caa1687bc77b5f2107cc043d09c4d8"D

# Some time later after retry - successful
Dec 14 21:53:32 host.domain dockerd[20062]: time="2022-12-14T21:53:32.060037672Z" level=debug msg="Removing volume reference: driver local, name a88def775aa002b128e8e27d9a8f311bebfaa78404be73c7b3c288daa9167dbe"

Reproduce

1. Create container
2. List dangling volumes
3. Delete dangling volumes
4. Start container

Expected behavior

Only dangling containers should be returned and they should be allowed to be removed (assuming no additional interactions with the daemon in the meanwhile).

docker version

Client:
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.15.14
 Git commit:        f0df350
 Built:             Wed Nov 17 03:05:36 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.14
  Git commit:       b0f5bc3
  Built:            Wed Nov 17 03:06:14 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0
  GitCommit:        84113eef6fc27af1b01b3181f31bbaf708715301
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: (redacted)
  Log: (redacted)
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: 84113eef6fc27af1b01b3181f31bbaf708715301
 init version: de40ad0
 Security Options:
  (redacted)
 Kernel Version: 5.4.225-139.416.amzn2int.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.36GiB
 Name: (redacted)
 ID: (redacted)
 Docker Root Dir: /local/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  (redacted)
 Live Restore Enabled: false

Additional Info

No response

Start by tracing the volume list endpoint with the dangling filter and the volume delete endpoint, focusing on how container references are recorded and checked. Reproduce the listed sequence with a stopped container, then verify that the dangling list excludes in-use volumes and that returned volumes can be deleted without a retry.