oxidecomputer / oxidecomputer/omicron
Many tests flaky after #8603
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
After commit cc1aba6a4f3649bc0ed699de9199456dd6fd434f (#8603), running large batch of tests with high parallelism (e.g., cargo nextest run --no-fail-fast on a 32 core machine) often produces many failures (50-90, depending on the run) of the form
expected to find inventory collection: TimedOut(30.057651591s)
This is probably also the cause of #8738. It does not appear as often in CI (probably because of the much lower degree of parallelism), but does occasionally, e.g., #8755, #8658. Running locally with reduced parallelism via -j also reduces the failures: with -j4 or -j8, the failures mostly vanish, but they appear again at -j16 and higher.
Here's a sample failure log. It may not actually be very useful for debugging, though, because of #1248.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce with cargo nextest run --no-fail-fast at varying -j values, using the test_blueprint_edit integration failure as a reference. Compare the behavior around commit #8603 and inspect the timeout logs, while accounting for the debugging limitation noted in #1248. Done means the high-parallelism batch no longer produces these inventory-collection timeouts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100