rust-lang / rust-lang/libc

A heisenbug in aarch64-linux-android CI

Open
#2,081 1 comment 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-CI C-bug E-medium O-android O-arm
Dominant language
Rust
Stars
2.6k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
69

Description

CI for that target is currently disabled due to this issue.

Hello, traveler.

I have stories to tell of strange bugs from unseen lands. Monsters which flicker in and out of sight, vanishing in the blink of an eye, hidden in virtual domains that themselves coalesce into being and then topple back into sand.

This primarily affects the aarch64-linux-android CI but it also affects the sparc-unknown-linux-gnu CI.
This was found at least as far back at #1851.
Several error logs are quoted into #2078.
The associated errors seemed to occur more often during the middle of the day rather than at night, so it may be an occurrence of a "noisy neighbor" problem in the virtual environment. It may also be an issue with QEMU.

This happens near the the cmsg tests, but can also happen during errqueue tests, the primary messages are

cmsg

stderr ---
thread 'main' panicked at 'No option 'h' defined', /cargo/registry/src/github.com-1ecc6299db9ec823/getopts-0.2.21/src/lib.rs:799:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

thread 'main' panicked at 'failed to find successful test run', /tmp/runtest.rs:45:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: test failed, to rerun pass '--test cmsg'

errqueue

stderr ---
thread 'main' panicked at 'No option 'h' defined', /cargo/registry/src/github.com-1ecc6299db9ec823/getopts-0.2.21/src/lib.rs:799:21
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

thread 'main' panicked at 'failed to find successful test run', /tmp/runtest.rs:46:9
stack backtrace:
   0: std::panicking::begin_panic
   1: runtest::main::{{closure}}
   2: core::option::Option<T>::unwrap_or_else
   3: runtest::main
   4: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: test failed, to rerun pass '--test errqueue'

In 5c07fcfd1c1333c5e6632ec96e169bd832ac5b57 we decided to add, and for the moment keep, RUST_BACKTRACE=1 at appropriate points in CI so as to hopefully surface a more useful backtrace when this triggers again. In fcae5a7a1a65b56d89f38cafdc59659d1f756620 we decided to cut aarch64-linux-android out of CI.

But we could do no more than place that warding to blockade that cyclopean archway. The monster still lives, and in time, it may come again, between the blinks of our strange existences.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the cmsg and errqueue tests and the error logs quoted in #2078, then compare the aarch64-linux-android and sparc-unknown-linux-gnu CI runs. Review the changes in commits 5c07fcfd1c1333c5e6632ec96e169bd832ac5b57 and fcae5a7a1a65b56d89f38cafdc59659d1f756620. Done means identifying the intermittent failure's cause, applying a reliable fix, and determining whether the disabled CI target can be restored.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, rust
Domain
ci-cd, operating-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.