containers / containers/fuse-overlayfs

SQLite rpmdb corruption when building centos-bootc images on btrfs

Open
#459 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
691
Forks
109
Avg merge
5d 7h
Merged PRs (30d)
2

Description

### Issue Description

Building container images on top of `quay.io/centos-bootc/centos-bootc:stream9` (an ostree-exported image with 65+ layers) using fuse-overlayfs corrupts the SQLite rpmdb when installing a significant number of packages. The same build with native kernel overlay produces a clean rpmdb.

The corruption manifests as `btreeInitPage() returns error code 11` on pages at the boundary where new data was written to the database (pages 9168+ out of 10065 total). The base image's rpmdb has ~9165 pages; the corruption starts exactly where the new package data begins.

The `rpmdb.sqlite` file in the base image has 3 hardlinks (to `/usr/share/rpm/rpmdb.sqlite`, `/usr/lib/sysimage/rpm-ostree-base-db/rpmdb.sqlite`, and an ostree object). This may be relevant to the corruption mechanism.

### Environment

- **OS**: Fedora 43 (kernel 6.19.13-200.fc43.x86_64)
- **fuse-overlayfs**: 1.13-4.fc43.x86_64
- **podman**: 5.8.2-1.fc43.x86_64
- **Backing filesystem**: btrfs

### Steps to reproduce

```bash
cat > /tmp/Containerfile << 'EOF'
FROM quay.io/centos-bootc/centos-bootc:stream9
RUN dnf -y install epel-release epel-next-release && \
dnf -y install audit beakerlib cloud-init createrepo_c \
dnf-plugins-core glibc-langpack-en koji rsync
EOF

# Build forcing fuse-overlayfs
podman --root=/tmp/fuse-test/storage --runroot=/tmp/fuse-test/runroot \
--storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs \
build --no-cache --network=host \
-f /tmp/Containerfile /tmp/ -t test-fuse:cs9

# Verify rpmdb
podman --root=/tmp/fuse-test/storage --runroot=/tmp/fuse-test/runroot \
--storage-opt overlay.mount_program=/usr/bin/fuse-overlayfs \
run --rm test-fuse:cs9 rpm --verifydb
```

### Results

With **fuse-overlayfs**:
```
error: verify: *** in database main ***
Page 10055: btreeInitPage() returns error code 11
Page 10006: btreeInitPage() returns error code 11
Page 9995: btreeInitPage() returns error code 11
Page 9986: btreeInitPage() returns error code 11
...
(hundreds of corrupt pages from 9168 to 10063)
```

With **native kernel overlay** (same host, same packages, same Containerfile):
```
(no output - rpmdb is clean)
```

### Verification that fuse-overlayfs is the cause

On the same F43 host (btrfs), all of the following produce a **clean** rpmdb:
- Build with native kernel overlay (default for rootful podman)
- Build with `--storage-opt overlay.mount_program=` — forces native overlay

Only builds using fuse-overlayfs produce the corruption.

### Key observations

- The base image (`quay.io/centos-bootc/centos-bootc:stream9`) rpmdb is clean — verified with `rpm --verifydb`
- The base image is an ostree export with **65+ overlay layers**
- The `rpmdb.sqlite` has **3 hardlinks** across different paths in the base image
- RPM 4.16 (CS9) switches the journal mode from `delete` to `wal` when writing, creating `-shm` and `-wal` files
- Corruption is in the **newly written pages only** (pages 9168+), not in the copied-up original data
- Installing a single small package (e.g., `tree`) does NOT trigger corruption — it requires a larger install that extends the database significantly
- The issue reproduces **every time**, not intermittently

### Related issues

- #333 — Image corruption when using fuse-overlayfs instead of overlayfs (same class of bug, fixed in PR #335 / fuse-overlayfs 1.9, but this is a different manifestation on current 1.13)
- #306 — Fuse-overlayfs produces filesystems considered corrupted by kernel overlayfs
- #192 — Incorrect number of links reported by stat when making hardlink
- https://github.com/containers/podman/issues/4764 — No good choice for storage driver on backing btrfs

### Workaround

Disable fuse-overlayfs in `/etc/containers/storage.conf`:
```toml
[storage.options.overlay]
mount_program = ""
mountopt = "nodev"
```
Then run `podman system reset -f` to clear existing storage. This forces native kernel overlay which does not have the corruption issue.

Generated-by: Claude Code

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the provided Containerfile build on btrfs with fuse-overlayfs and compare it with native kernel overlay, then run rpm --verifydb. Inspect the hardlinked rpmdb.sqlite and its WAL files around the newly written pages. Done means the reproduced build produces a clean rpmdb, ideally with regression coverage for the hardlink and layered-image case.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust, sqlite
Domain
databases, operating-systems, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.