oxidecomputer / oxidecomputer/omicron
CI test flake: helios worker timeout (?)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Two instances of this failure on different branches:
- https://github.com/oxidecomputer/omicron/pull/4959/checks?check_run_id=21248439235
- https://github.com/oxidecomputer/omicron/pull/5018/checks?check_run_id=21337591397
In both cases, we failed the helios build-and-test CI job (other jobs passed), and in both cases the job log appears to indicate that we stopped getting output from either nextest or something above it in the buildomat / worker system. The tail of the job log from the second:
2024-02-07T21:46:42.918Z PASS [ 5.334s] nexus-db-queries db::collection_detach::test::test_detach_once
2024-02-07T21:46:43.197Z PASS [ 37.216s] bootstore::v0-fsm-proptest-rack-coordinator run
2024-02-07T21:46:44.299Z PASS [ 5.281s] nexus-db-queries db::collection_detach::test::test_detach_while_already_detached
2024-02-07T21:46:44.422Z PASS [ 5.247s] nexus-db-queries db::collection_detach::test::test_detach_without_update_filter
2024-02-07T21:46:45.049Z PASS [ 5.058s] nexus-db-queries db::collection_detach_many::test::test_detach_deleted_resource
2024-02-07T21:46:45.067Z PASS [ 0.028s] nexus-db-queries db::collection_detach_many::test::test_verify_query
2024-02-08T05:10:45.220Z job duration 28813 exceeds 28800 seconds; aborting
2024-02-08T05:10:45.630Z worker failed without completing job
2024-02-08T05:10:45.632Z task 4 was incomplete, marked failed
There's a muliple-hour jump between the last line we got from nextest and when the timeout failed the job. This doesn't look like any individual test failed or timed out; nextest itself catches those (and continues to print output for slow tests as they run). There are also no test logs collected on these jobs, so I'm not sure how to debug this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the two linked helios build-and-test check logs and compare the final nextest output with the multi-hour gap before the worker timeout. Determine whether the stall is in nextest or the buildomat/worker system, then verify the fix or added diagnostics against the same CI job.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100