A heisenbug in aarch64-linux-android CI
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 2.6k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 69
Description
CI for that target is currently disabled due to this issue.
Hello, traveler.
I have stories to tell of strange bugs from unseen lands. Monsters which flicker in and out of sight, vanishing in the blink of an eye, hidden in virtual domains that themselves coalesce into being and then topple back into sand.
This primarily affects the aarch64-linux-android CI but it also affects the sparc-unknown-linux-gnu CI.
This was found at least as far back at #1851.
Several error logs are quoted into #2078.
The associated errors seemed to occur more often during the middle of the day rather than at night, so it may be an occurrence of a "noisy neighbor" problem in the virtual environment. It may also be an issue with QEMU.
This happens near the the cmsg tests, but can also happen during errqueue tests, the primary messages are
cmsg
stderr ---
thread 'main' panicked at 'No option 'h' defined', /cargo/registry/src/github.com-1ecc6299db9ec823/getopts-0.2.21/src/lib.rs:799:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' panicked at 'failed to find successful test run', /tmp/runtest.rs:45:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: test failed, to rerun pass '--test cmsg'
errqueue
stderr ---
thread 'main' panicked at 'No option 'h' defined', /cargo/registry/src/github.com-1ecc6299db9ec823/getopts-0.2.21/src/lib.rs:799:21
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread 'main' panicked at 'failed to find successful test run', /tmp/runtest.rs:46:9
stack backtrace:
0: std::panicking::begin_panic
1: runtest::main::{{closure}}
2: core::option::Option<T>::unwrap_or_else
3: runtest::main
4: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: test failed, to rerun pass '--test errqueue'
In 5c07fcfd1c1333c5e6632ec96e169bd832ac5b57 we decided to add, and for the moment keep, RUST_BACKTRACE=1 at appropriate points in CI so as to hopefully surface a more useful backtrace when this triggers again. In fcae5a7a1a65b56d89f38cafdc59659d1f756620 we decided to cut aarch64-linux-android out of CI.
But we could do no more than place that warding to blockade that cyclopean archway. The monster still lives, and in time, it may come again, between the blinks of our strange existences.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the cmsg and errqueue tests and the error logs quoted in #2078, then compare the aarch64-linux-android and sparc-unknown-linux-gnu CI runs. Review the changes in commits 5c07fcfd1c1333c5e6632ec96e169bd832ac5b57 and fcae5a7a1a65b56d89f38cafdc59659d1f756620. Done means identifying the intermittent failure's cause, applying a reliable fix, and determining whether the disabled CI target can be restored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, rust
- Domain
- ci-cd, operating-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100