oxidecomputer / oxidecomputer/omicron
ci: helios-deploy: 300 seconds may not be enough
Open
Nobody has claimed this yet.
Test Flake
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
A recent deploy job in CI failed because nexus had not come up in 300 seconds.
Looking at the logs we can see
2024-07-12T23:15:18.972Z sled agent starts to initialize CRDB (sled agent log)
2024-07-12T23:15:23.765Z ci test declares failure at 300 seconds (deploy job log)
So it does look like the system was on its way to coming up but was declared a failure just after CRDB initialization started.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked helios-deploy run and its deploy job log, then locate the CI entry point that declares failure after 300 seconds. Check how CRDB initialization and nexus readiness are timed; done means a deployment that is still progressing after 300 seconds is not failed prematurely, while genuinely unavailable nexus deployments still fail.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ci-cd
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100