containers / containers/fuse-overlayfs

Intermittent silent data corruption in large SQLite file rewritten through fuse-overlayfs 1.16 (rootless buildah on GitHub Actions runners)

Open
#475 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
691
Forks
109
Avg merge
5d 7h
Merged PRs (30d)
2

Description

## Summary

Container image builds that rewrite a large (~90 MB) SQLite database inherited from a lower layer intermittently produce silently corrupted databases when buildah's storage uses `overlay.mount_program=fuse-overlayfs`. The identical build through kernel-native overlayfs has never corrupted in our testing (10 for 10 clean), while the fuse-overlayfs path corrupts intermittently in time-clustered bursts (7 corrupt builds observed so far, including 4 consecutive daily production builds).

## Environment

- fuse-overlayfs 1.16, binary at `/usr/local/bin/fuse-overlayfs` on GitHub-hosted `ubuntu-24.04` runners (image 20260810.271.1 and 20260816.277)
- buildah 1.33.7, rootless, driver `overlay` with `STORAGE_OPTS=overlay.mount_program=/usr/local/bin/fuse-overlayfs` (set automatically by redhat-actions/buildah-build when the binary exists)
- Kernel 6.17.0-1022-azure, Ubuntu 24.04.4
- Workload: `dnf install`/`dnf remove` inside a Fedora 44 bootc image build (rpm 6.0.2, sqlite-libs 3.51.2). rpm's database is `/usr/lib/sysimage/rpm/rpmdb.sqlite`, roughly 90 MB, WAL mode during the transaction. It is inherited from the base image, so the write path is copy-up followed by many small page writes.

## Symptom

The built image's rpm database is corrupt. `rpm -qa` reports:

```
error: rpmdbNextIterator: skipping h# 2142 region trailer: BAD, tag 1869873266 type 1869575168 offset -1919905652 count 7499631
error: SELECT hnum, blob FROM 'Packages': 11: database disk image is malformed
```

`PRAGMA integrity_check` on the extracted file reports about 100 errors: `btreeInitPage() returns error code 11` in several trees, "2nd reference to page", "invalid page number 218103808", "Rowid out of order", and roughly 90 "Page never used" orphans. Hexdumping a corrupted page of the `Dirnames` b-tree shows content that belongs somewhere else entirely (a package's Requires strings such as `libstdc++.so.6(GLIBCXX_3.4.19)(64bit)`). In other words, pages were written to the wrong offsets or into the wrong file region. Each occurrence corrupts different pages.

A possibly relevant detail: the same dnf transaction writes more than one SQLite database (the rpmdb plus libdnf5 state). SQLite keys its POSIX lock and fd bookkeeping by (st_dev, st_ino), so non-unique st_ino values across files would alias that state between two open databases. The fuse-overlayfs 1.17 release notes mention "st_ino needs to be unique for all inodes on a device"; these runners ship 1.16.

## Reproduction statistics

All builds are byte-identical inputs (same Containerfile, same base image digest, same buildah command line):

| Storage path | Corrupt / total | Notes |
| --- | --- | --- |
| fuse-overlayfs, GitHub runners | 3 / 17 | all 3 failures within one 5-minute window |
| fuse-overlayfs, daily production builds at ~11:10 UTC | 4 / 4 | consecutive days, 2026-08-15 through 2026-08-19 |
| native kernel overlayfs, same runners, same build | 0 / 7 | |
| native kernel overlayfs, local rootless podman (Fedora 44) | 0 / 3 | |

The flip from always-clean to frequently-corrupt in production coincided exactly with GitHub's runner image 20260810 adding the fuse-overlayfs binary (the previous 20260720 image did not have it, so builds used native overlay and never corrupted). Nothing else changed: same action commit since June, byte-identical `buildah bud` command line, and identical rpm/sqlite/dnf versions inside the image on the last-clean and first-corrupt days.

Public runs (jobs named `VERDICT: CORRUPT`/`CLEAN` in the "Check rpmdb health" step):

- Corrupt, 3 of 3 in one run: https://github.com/OptimoSupreme/Tetra-Tailored/actions/runs/32287596040
- Clean repeats of the same configuration later the same day: https://github.com/OptimoSupreme/Tetra-Tailored/actions/runs/32289899695 and https://github.com/OptimoSupreme/Tetra-Tailored/actions/runs/32290874355
- Corrupt production build (2026-08-15): https://github.com/OptimoSupreme/Tetra-Tailored/actions/runs/31881364889

The corrupt published images are also public if useful: `ghcr.io/optimosupreme/tetra-tailored:balrog-20260815` (and later daily tags) show the corruption; `balrog-20260814` and earlier are clean.

## What I can provide

Corrupt rpmdb.sqlite samples, full integrity_check output, hexdumps of corrupted pages, and links to any of the runs above. Happy to run further instrumented builds since the daily production job reproduces this reliably.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the corruption with fuse-overlayfs 1.16 using the provided buildah and GitHub Actions configuration, then compare it with native overlayfs and fuse-overlayfs 1.17. Review the supplied corrupt rpmdb.sqlite samples, integrity-check output, and hexdumps alongside the st_ino detail. Done means the reproduction is explained and the affected build path no longer produces corrupt databases.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sqlite
Domain
databases, infrastructure, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.