test flakes tracker
- Dominant language
- Rust
- Stars
- 2.3k
- Forks
- 230
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 38
Description
# CI Flakes Tracker
This tracks CI failures, usually in the merge queue.
## COPR repo unavailable
```
Error: Failed to download metadata for repo
'copr:copr.fedorainfracloud.org:group_CoreOS:continuous':
Cannot download repomd.xml: All mirrors were tried
```
The COPR continuous repo is transiently unavailable during the container build,
causing `dnf` to fail. Happens specifically on centos-10 jobs that pull from
the CoreOS COPR. Retry would likely succeed.
## quay.io 502
**Error:**
```
error: Installing to filesystem: Creating ostree deployment: Pulling:
Importing: Unencapsulating base: failed to invoke method GetBlob:
fetching blob: received unexpected HTTP status: 502 Bad Gateway
```
The in-VM install test hit a transient 502 from the container registry while
pulling an image. Pure network flake.
## GHA runner killed mid-TMT
**Symptom:** Job ran unit tests successfully, TMT step started but produced no
conclusion after 2h15m. No artifact uploaded. Run marked failed with no logged
error. Runner appears to have been killed externally (OOM or host issue).
## `test_container_write_derive` assertion failure
**Frequency:** Once (fedora-44, ostree, xfs, grub, bls, unsealed — Unit and container integration tests)
**Error:**
```
assertion `left == right` failed
left: "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"
right: "newderivedfile v0"
at crates/ostree-ext/tests/it/main.rs:1713
```
File written into a derived container was read back as all null bytes. The same
test passed in `fedora-43, ostree` in the same queue batch. Likely a race
condition or tmpfs/storage timing issue on the fedora-44 runner; not seen
before or since.
## Docs
Basically `cargo binstall` tries to fetch binaries from GitHub releases, but because we aren't passing in a `GH_TOKEN` into that inner job we can get rate limited. When that happens, `cargo binstall` falls back to trying to compile from source, which can then fail due to missing dependencies.
## Package expiration
Our workflow splits package jobs from integration tests, if a PR sticks around too long the package artifacts expire, meaning *all* workflows need to be retried if a flake happens. I think we could extend package expiry to a week or potentially even longer.
Contributor guide
Assessment
This issue has not been assessed yet.