oxidecomputer / oxidecomputer/omicron
Uncover why some progenitor requests hit timeout, and others return immediately
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
In the context of https://github.com/oxidecomputer/omicron/issues/8756, we noticed tests hitting collection timeouts. Within https://github.com/oxidecomputer/omicron/issues/8756#issuecomment-3152459100, it's clear that "sometimes, especially under load/concurrent testing, we hit the max timeout when contacting services".
We can mitigate the flake by lowering these timeouts, but this leaves the unanswered questions of:
- What causes us to hit these timeouts in some, but not all cases?
- What services are susceptible to hitting these timeouts?
- Can we track the duration of inventory collection in a benchmark, to mitigate this issue in the future?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with issue #8756 and its linked comment, then trace the inventory collection requests that can hit the maximum timeout under concurrent load. Identify the affected services and causes, and determine whether collection duration can be tracked in a benchmark so the timeout flake can be monitored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100