`tar: .: file changed as we read it` with `fuse-overlayfs`

/kind bug

**Description**

I've recently migrated CI to use `fuse-overlayfs` due [to an issue with](https://github.com/containers/podman/issues/16541#issuecomment-1351195823) with native overlay, and after some time it started sporadically failing with error in title. I am not completely sure fuse-overlayfs is the cause, because of an unrelated podman bug which I'll report separately, but that was the only change, so it likely is.

Problem comes down to this:

```
λ podman run --rm ubuntu:22.04 sh -c "tar -cf /tmp/flash.tar --one-file-system --exclude=./* -C / ."
tar: .: file changed as we read it
```

To dispel any doubts the `tar` command is valid: you can test it on a host system, it will succeed. The `tar` is being asked to enter `/` dir with `-C /`, then to save everything from the current dir that is not `--exclude`ed into `/tmp/flash.tar`. However, for the purposes of a "minimal testcase" the `--exclude` command excludes everything, so `/tmp/flash.tar` will only contain an empty `./` dir.

The `tar` error comes down to mtime of a `/` being changed while it's running. Why does that happen, I haven't yet found. I did find though that various odd combinations make that not reproduce. Some examples in the **workarounds** section below.

The error happens on Ubuntu and Arch and with different containers. I'm reporting Arch as it's my working system and is easily accessible.

**Known Workarounds**

1.
```
λ podman run --rm ubuntu:22.04 sh -c "tar -cf /tmp/flash.tar --one-file-system --exclude=./* -C / .; tar -cf /tmp/flash.tar --one-file-system --exclude=./* -C / ."
tar: .: file changed as we read it
```
Here you can see the `tar` command is being called twice, but the error only happens the first time.
2.
```
λ podman run --rm ubuntu:22.04 sh -c "ls / >/dev/null && tar -cf /tmp/flash.tar --one-file-system --exclude=./* -C / ."
```
No error. The only thing changed is we call a `ls /`, which alone fixes the problem.
3.
```
λ podman run --rm -v /tmp/:/tmp ubuntu:22.04 sh -c "touch / && tar -cf /tmp/flash.tar --one-file-system --exclude=./* -C / ."
```
No error. There are 2 changes:
* the `tar` destination dir belongs to host through `-v …`
* there's a `touch /` command before calling `tar`

Doing any of them alone will result in the error, that only works as a combination.

**Steps to reproduce the issue *(in terms of terminal commands)*:**

```bash
λ podman run --rm ubuntu:22.04 sh -c "tar -cf /tmp/flash.tar --one-file-system --exclude=./* -C / ."
tar: .: file changed as we read it
```

**Describe the results you received:**

An error:
tar: .: file changed as we read it

**Describe the results you expected:**

No output and zero exit code.

**Output of `podman version`:**

```
λ podman version
Client: Podman Engine
Version: 4.3.1
API Version: 4.3.1
Go Version: go1.19.3
Git Commit: 814b7b003cc630bf6ab188274706c383f9fb9915-dirty
Built: Mon Nov 21 01:32:45 2022
OS/Arch: linux/amd64
```

**Output of `podman info`:**

Details

```
λ podman version
Client: Podman Engine
Version: 4.3.1
API Version: 4.3.1
Go Version: go1.19.3
Git Commit: 814b7b003cc630bf6ab188274706c383f9fb9915-dirty
Built: Mon Nov 21 01:32:45 2022
OS/Arch: linux/amd64
[20.12.2022-17:28:29] constantine@constantine-N61Ja /tmp ‹node-› ‹›
λ podman info
host:
arch: amd64
buildahVersion: 1.28.0
cgroupControllers:
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.1.5-1
path: /usr/bin/conmon
version: 'conmon version 2.1.5, commit: c9f7f19eb82d5b8151fc3ba7fbbccf03fdcd0325'
cpuUtilization:
idlePercent: 62.71
systemPercent: 9.31
userPercent: 27.98
cpus: 4
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: constantine-N61Ja
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 165536
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 165536
size: 65536
kernel: 6.0.11-zen1-1-zen
linkmode: dynamic
logDriver: journald
memFree: 893280256
memTotal: 8225759232
networkBackend: cni
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 1.7.2-1
path: /usr/bin/crun
version: |-
crun version 1.7.2
commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
version: |-
slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.4
swapFree: 3161047040
swapTotal: 7918841856
uptime: 272h 34m 42.00s (Approximately 11.33 days)
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- docker.io
- registry.fedoraproject.org
- quay.io
- registry.access.redhat.com
- registry.centos.org
store:
configFile: /home/constantine/.config/containers/storage.conf
containerStore:
number: 9
paused: 0
running: 2
stopped: 7
graphDriverName: overlay
graphOptions:
overlay.mount_program:
Executable: /usr/bin/fuse-overlayfs
Package: /usr/bin/fuse-overlayfs is owned by fuse-overlayfs 1.10-1
Version: |-
fusermount3 version: 3.12.0
fuse-overlayfs: version 1.10
FUSE library version 3.12.0
using FUSE kernel interface version 7.31
graphRoot: /home/constantine/.local/share/containers/storage
graphRootAllocated: 991614205952
graphRootUsed: 726417346560
graphStatus:
Backing Filesystem: btrfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 30
runRoot: /run/user/1000/containers
volumePath: /home/constantine/.local/share/containers/storage/volumes
version:
APIVersion: 4.3.1
Built: 1668983565
BuiltTime: Mon Nov 21 01:32:45 2022
GitCommit: 814b7b003cc630bf6ab188274706c383f9fb9915-dirty
GoVersion: go1.19.3
Os: linux
OsArch: linux/amd64
Version: 4.3.1
```

**Package info:**

```
λ pacman -Qi podman
Name : podman
Version : 4.3.1-2
Description : Tool and library for running OCI-based containers in pods
Architecture : x86_64
URL : https://github.com/containers/podman
Licenses : Apache
Groups : None
Provides : None
Depends On : catatonit conmon containers-common crun iptables libdevmapper.so=1.02-64 libgpgme.so=11-64 libseccomp.so=2-64 slirp4netns
Optional Deps : apparmor: for AppArmor support
btrfs-progs: support btrfs backend devices [installed]
cni-plugins: for an alternative container-network-stack implementation [installed]
podman-compose: for docker-compose compatibility
podman-docker: for Docker-compatible CLI
Required By : None
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 66.95 MiB
Packager : David Runge
Build Date : Пн 21 ноя 2022 01:32:45
Install Date : Ср 07 дек 2022 23:38:35
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature
```

**Have you tested with the latest version of Podman and have you checked [the Podman Troubleshooting Guide](https://github.com/containers/podman/blob/main/troubleshooting.md)?**

Yes

Start by reproducing the reported podman run command with fuse-overlayfs 1.10, then compare the listed workarounds and storage configuration. Trace the mtime change for the container root while tar runs; done means identifying the fuse-overlayfs cause and preventing the spurious tar error without breaking the documented cases.

Written by the indexing model from the issue text.