oxidecomputer / oxidecomputer/omicron
something fishy is happening in the `deploy.sh` job during the switch zone check
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Our check that the switch zone is up was, for whatever reason, hanging for approximately 195 seconds on each iteration. During this time according to the 35 MiB of Dendrite logs, Dendrite was handling requests normally.
#9767 slaps a band-aid over the problem to prevent it from kerploding the CI queue length but doesn't get at the root cause.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked Buildomat run details and inspect the deploy.sh switch-zone-up check alongside the Dendrite logs described in the issue. Review #9767 to understand the temporary mitigation; done means identifying why each check waits about 195 seconds and addressing the root cause so the deploy job no longer stalls.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- shell
- Domain
- ci-cd, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100