oxidecomputer / oxidecomputer/omicron
Installinator download taking substantially longer and sometimes stuck at 100%
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
A recent rack2 mupdate from a branch commit (3a90a98/4b23a73 which has caught up on omicron main commits) to af3b89f/4b23a73 took 1hr 11mins to complete updating 9 of the 11 sleds. 2 sleds (cubby 8 & 9) were stuck in installinator 100% downloading. I was able to retry their update after aborting/clearing the update on wicket, w/o having to restart any service.
@leftwo has seen installinator download % being reset when doing mupdates on a racklet. The issue was tracked in https://github.com/oxidecomputer/stlouis/issues/844 as the sled was hung. I didn't observe that in my case.
I'm filing this ticket to track the wicket and mgs logs gathered from the rack2 mupdate round, plus any new information we may find in the next mupdate (logs are placed under /staff/core/omicron-<ticket#>).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the wicket and MGS logs gathered from the rack2 mupdate, stored under /staff/core/omicron-<ticket#>, and compare them with the installinator download behavior on cubby 8 and 9. Check the next mupdate for reproducible evidence and compare with the referenced stlouis issue 844. Done means identifying the cause of the stalled or slow downloads and defining the corrective change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100