bootc-dev / bootc-dev/bootc

test flakes tracker

Open
#2,177 21 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2.3k
Forks
230
Avg merge
3d 12h
Merged PRs (30d)
38

Description

# CI Flakes Tracker

This tracks CI failures, usually in the merge queue.

## COPR repo unavailable

```
Error: Failed to download metadata for repo
'copr:copr.fedorainfracloud.org:group_CoreOS:continuous':
Cannot download repomd.xml: All mirrors were tried
```
The COPR continuous repo is transiently unavailable during the container build,
causing `dnf` to fail. Happens specifically on centos-10 jobs that pull from
the CoreOS COPR. Retry would likely succeed.

## quay.io 502

**Error:**
```
error: Installing to filesystem: Creating ostree deployment: Pulling:
Importing: Unencapsulating base: failed to invoke method GetBlob:
fetching blob: received unexpected HTTP status: 502 Bad Gateway
```
The in-VM install test hit a transient 502 from the container registry while
pulling an image. Pure network flake.

## GHA runner killed mid-TMT

**Symptom:** Job ran unit tests successfully, TMT step started but produced no
conclusion after 2h15m. No artifact uploaded. Run marked failed with no logged
error. Runner appears to have been killed externally (OOM or host issue).

## `test_container_write_derive` assertion failure

**Frequency:** Once (fedora-44, ostree, xfs, grub, bls, unsealed — Unit and container integration tests)
**Error:**
```
assertion `left == right` failed
left: "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"
right: "newderivedfile v0"
at crates/ostree-ext/tests/it/main.rs:1713
```
File written into a derived container was read back as all null bytes. The same
test passed in `fedora-43, ostree` in the same queue batch. Likely a race
condition or tmpfs/storage timing issue on the fedora-44 runner; not seen
before or since.

## Docs

Basically `cargo binstall` tries to fetch binaries from GitHub releases, but because we aren't passing in a `GH_TOKEN` into that inner job we can get rate limited. When that happens, `cargo binstall` falls back to trying to compile from source, which can then fail due to missing dependencies.

## Package expiration

Our workflow splits package jobs from integration tests, if a PR sticks around too long the package artifacts expire, meaning *all* workflows need to be retried if a flake happens. I think we could extend package expiry to a week or potentially even longer.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.