A docker image is not removed from disk after enabling userns-remap, creating an image and removing the image

Description

A docker image is not removed from disk after enabling userns-remap, creating an image and removing the image

Reproduce

1. Create a dockuser user account on your docker host
2. Stop the docker daemon
3. Configure dockerd for userns-remap

{
    "userns-remap": "dockuser"
}

4. Start the docker daemon
5. Build a docker image
6. Run a container using that docker image
7. Stop the container
8. Remove the container
9. Remove the built docker image

[user@host ~]$ docker images -a
REPOSITORY            TAG       IMAGE ID       CREATED              SIZE
ubuntu_systemd_test   local     0a55bc64c86d   About a minute ago   2.27GB

[user@host ~]$ df -h /var/lib/docker
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        51G   46G  3.8G  93% /

[user@host ~]$ docker image rm 0a55bc64c86d
Untagged: ubuntu_systemd_test:local
Deleted: sha256:0a55bc64c86deb501a97534bcd5a555e4de35a31a3b5f58139c39d9623bd9d22

[user@host ~]$ sync

[user@host ~]$ df -h /var/lib/docker
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        51G   46G  3.8G  93% /

Expected behavior

The diskspace should've been reclaimed but it wasn't. I found out that the image was written under /var/lib/docker/262144.262144/overlay2/ Those are the subuid and subgid for the dockuser user account.

docker version

Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:37:21 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:43 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  userns
  cgroupns
 Kernel Version: 6.4.12-100.fc37.x86_64
 Operating System: Fedora Linux 37 (KDE Plasma)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.731GiB
 Name: ac
 ID: 57d8675c-0448-4386-9b41-92564aee0180
 Docker Root Dir: /var/lib/docker/262144.262144
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

No source file or test is named. Start by reproducing the issue with userns-remap and inspect the overlay2 storage under the Docker Root Dir after image removal; done means the image's disk space is reclaimed.