containers / containers/fuse-overlayfs
SQLite rpmdb corruption when building centos-bootc images on btrfs
- Dominant language
- Rust
- Stars
- 691
- Forks
- 109
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 2
Description
### Issue Description
Building container images on top of `quay.io/centos-bootc/centos-bootc:stream9` (an ostree-exported image with 65+ layers) using fuse-overlayfs corrupts the SQLite rpmdb when installing a significant number of packages. The same build with native kernel overlay produces a clean rpmdb.
The corruption manifests as `btreeInitPage() returns error code 11` on pages at the boundary where new data was written to the database (pages 9168+ out of 10065 total). The base image's rpmdb has ~9165 pages; the corruption starts exactly where the new package data begins.
The `rpmdb.sqlite` file in the base image has 3 hardlinks (to `/usr/share/rpm/rpmdb.sqlite`, `/usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite`, and an ostree object). This may be relevant to the corruption mechanism.
### Environment
- **OS**: Fedora 43 (kernel 6.19.13-200.fc43.x86_64)
- **fuse-overlayfs**: 1.13-4.fc43.x86_64
- **podman**: 5.8.2-1.fc43.x86_64
- **Backing filesystem**: btrfs
### Steps to reproduce
```bash
cat > /tmp/Containerfile << 'EOF'
FROM quay.io/centos-bootc/centos-bootc:stream9
RUN dnf -y install epel-release epel-next-release && \
dnf -y install audit beakerlib cloud-init createrepo_c \
dnf-plugins-core glibc-langpack-en koji rsync
EOF
# Build forcing fuse-overlayfs
podman --root=/tmp/fuse-test/storage --runroot=/tmp/fuse-test/runroot \
--storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs \
build --no-cache --network=host \
-f /tmp/Containerfile /tmp/ -t test-fuse:cs9
# Verify rpmdb
podman --root=/tmp/fuse-test/storage --runroot=/tmp/fuse-test/runroot \
--storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs \
run --rm test-fuse:cs9 rpm --verifydb
```
### Results
With **fuse-overlayfs**:
```
error: verify: *** in database main ***
Page 10055: btreeInitPage() returns error code 11
Page 10006: btreeInitPage() returns error code 11
Page 9995: btreeInitPage() returns error code 11
Page 9986: btreeInitPage() returns error code 11
...
(hundreds of corrupt pages from 9168 to 10063)
```
With **native kernel overlay** (same host, same packages, same Containerfile):
```
(no output - rpmdb is clean)
```
### Verification that fuse-overlayfs is the cause
On the same F43 host (btrfs), all of the following produce a **clean** rpmdb:
- Build with native kernel overlay (default for rootful podman)
- Build with `--storage-opt overlay.mount_program=` — forces native overlay
Only builds using fuse-overlayfs produce the corruption.
### Key observations
- The base image (`quay.io/centos-bootc/centos-bootc:stream9`) rpmdb is clean — verified with `rpm --verifydb`
- The base image is an ostree export with **65+ overlay layers**
- The `rpmdb.sqlite` has **3 hardlinks** across different paths in the base image
- RPM 4.16 (CS9) switches the journal mode from `delete` to `wal` when writing, creating `-shm` and `-wal` files
- Corruption is in the **newly written pages only** (pages 9168+), not in the copied-up original data
- Installing a single small package (e.g., `tree`) does NOT trigger corruption — it requires a larger install that extends the database significantly
- The issue reproduces **every time**, not intermittently
### Related issues
- #333 — Image corruption when using fuse-overlayfs instead of overlayfs (same class of bug, fixed in PR #335 / fuse-overlayfs 1.9, but this is a different manifestation on current 1.13)
- #306 — Fuse-overlayfs produces filesystems considered corrupted by kernel overlayfs
- #192 — Incorrect number of links reported by stat when making hardlink
- https://github.com/containers/podman/issues/4764 — No good choice for storage driver on backing btrfs
### Workaround
Disable fuse-overlayfs in `/etc/containers/storage.conf`:
```toml
[storage.options.overlay]
mount_program = ""
mountopt = "nodev"
```
Then run `podman system reset -f` to clear existing storage. This forces native kernel overlay which does not have the corruption issue.
Generated-by: Claude Code
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the provided Containerfile build on btrfs with fuse-overlayfs and compare it with native kernel overlay, then run rpm --verifydb. Inspect the hardlinked rpmdb.sqlite and its WAL files around the newly written pages. Done means the reproduced build produces a clean rpmdb, ideally with regression coverage for the hardlink and layered-image case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, rust, sqlite
- Domain
- databases, operating-systems, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100