oxidecomputer / oxidecomputer/omicron

Many tests flaky after #8603

Open
#8,756 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

After commit cc1aba6a4f3649bc0ed699de9199456dd6fd434f (#8603), running large batch of tests with high parallelism (e.g., cargo nextest run --no-fail-fast on a 32 core machine) often produces many failures (50-90, depending on the run) of the form

expected to find inventory collection: TimedOut(30.057651591s)

This is probably also the cause of #8738. It does not appear as often in CI (probably because of the much lower degree of parallelism), but does occasionally, e.g., #8755, #8658. Running locally with reduced parallelism via -j also reduces the failures: with -j4 or -j8, the failures mostly vanish, but they appear again at -j16 and higher.

Here's a sample failure log. It may not actually be very useful for debugging, though, because of #1248.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce with cargo nextest run --no-fail-fast at varying -j values, using the test_blueprint_edit integration failure as a reference. Compare the behavior around commit #8603 and inspect the timeout logs, while accounting for the debugging limitation noted in #1248. Done means the high-parallelism batch no longer produces these inventory-collection timeouts.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.