oxidecomputer / oxidecomputer/omicron

CI test flake: helios worker timeout (?)

Open
#5,027 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Two instances of this failure on different branches:

In both cases, we failed the helios build-and-test CI job (other jobs passed), and in both cases the job log appears to indicate that we stopped getting output from either nextest or something above it in the buildomat / worker system. The tail of the job log from the second:

2024-02-07T21:46:42.918Z	        PASS [   5.334s] nexus-db-queries db::collection_detach::test::test_detach_once
2024-02-07T21:46:43.197Z	        PASS [  37.216s] bootstore::v0-fsm-proptest-rack-coordinator run
2024-02-07T21:46:44.299Z	        PASS [   5.281s] nexus-db-queries db::collection_detach::test::test_detach_while_already_detached
2024-02-07T21:46:44.422Z	        PASS [   5.247s] nexus-db-queries db::collection_detach::test::test_detach_without_update_filter
2024-02-07T21:46:45.049Z	        PASS [   5.058s] nexus-db-queries db::collection_detach_many::test::test_detach_deleted_resource
2024-02-07T21:46:45.067Z	        PASS [   0.028s] nexus-db-queries db::collection_detach_many::test::test_verify_query
 
2024-02-08T05:10:45.220Z	job duration 28813 exceeds 28800 seconds; aborting
2024-02-08T05:10:45.630Z	worker failed without completing job
2024-02-08T05:10:45.632Z	task 4 was incomplete, marked failed

There's a muliple-hour jump between the last line we got from nextest and when the timeout failed the job. This doesn't look like any individual test failed or timed out; nextest itself catches those (and continues to print output for slow tests as they run). There are also no test logs collected on these jobs, so I'm not sure how to debug this.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the two linked helios build-and-test check logs and compare the final nextest output with the multi-hour gap before the worker timeout. Determine whether the stall is in nextest or the buildomat/worker system, then verify the fix or added diagnostics against the same CI job.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.