Directory with .wh..wh..opaque extension is presented inside image

Deleted directory with .wh..wh..opaque extension is presented inside image

I am writing a program that consequentially parses layers of image and tries to find binaries inside image. Because images can have in some layers deleted files and directories, I try to implement some merge approach to take a final state of the file system. But now I have some confusing situation with image that have layers with deleted directories, but this directory and all her content are presented in the final state.

The most interesting situation is that I can reproduce this behavior when building image on ubuntu:bionic or ubuntu:focal. But when build the image on virtual machine with ubuntu:xenial the same directory doesn't have .wh..wh..opaque extensions.

Steps to reproduce the issue:
I built some tests images to reproduce the issue:

1. For image building was using https://github.com/bitnami/bitnami-docker-kube-state-metrics/blob/1.9.4-debian-10-r13/1/debian-10/Dockerfile and images were built on 2 systems:

• ubuntu:bionic - stavrovskaya/prometeus:test_bionic
• ubuntu:xenial: stavrovskaya/prometeus:test_xenial

2. In stavrovskaya/prometeus:test_bionic the 3-d layer looks like this:

 'opt'
 'opt/bitnami'
 'opt/bitnami/.wh..wh..opq'
 'opt/bitnami/kube-state-metrics'
 'opt/bitnami/kube-state-metrics/bin'
 'opt/bitnami/kube-state-metrics/bin/kube-state-metrics'
 'opt/bitnami/kube-state-metrics/licenses'
 'opt/bitnami/kube-state-metrics/licenses/kube-state-metrics-1.9.4.txt'
 'tmp'
 'tmp/bitnami'
 'tmp/bitnami/.wh..wh..opq'
 'tmp/bitnami/pkg'
 'tmp/bitnami/pkg/cache'

In stavrovskaya/prometeus:test_xenial the same layer is:
'opt'
'opt/bitnami'
'opt/bitnami/kube-state-metrics'
'opt/bitnami/kube-state-metrics/bin'
'opt/bitnami/kube-state-metrics/bin/kube-state-metrics'
'opt/bitnami/kube-state-metrics/licenses'
'opt/bitnami/kube-state-metrics/licenses/kube-state-metrics-1.9.4.txt'
'tmp', 'tmp/bitnami'
'tmp/bitnami/pkg'
'tmp/bitnami/pkg/cache'

Interesting fact that official image bitnami/kube-state-metrics:1.9.4-debian-10-r13 equals by structure with stavrovskaya/prometeus:test_xenial

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:36 2021
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  scan: Docker Scan (Docker Inc., v0.12.0)

Server:
 Containers: 35
  Running: 7
  Paused: 0
  Stopped: 28
 Images: 89
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-166-generic
 Operating System: Ubuntu 18.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 31.29GiB
 Name: stavrovskaya-pc
 ID: WFM5:SY5Y:BJZC:MQNG:6O6M:2LEG:MJXU:V56S:HQVU:POEG:FHOD:7GF2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: stavrovskaya
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 172.240.0.0/16, Size: 24

Start by reproducing the layer listings with the referenced stavrovskaya/prometeus:test_bionic and test_xenial images on the stated Docker and overlay2 setup. Trace how image layers and .wh..wh..opq entries are merged, then verify that deleted directories do not appear in the final filesystem while preserving the expected behavior for both images.