oxidecomputer / oxidecomputer/omicron
sled-agent failed to notice propolis-server panicked
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
The propolis-server running an instance on rack2 panicked, but the instance state continued to be reported in the console as "Running". The instance_state_monitor task in sled-agent is responsible for observing state changes from propolis, which loops calling the instance_state_monitor endpoint. If propolis-server panics, then propolis will stop returning requests from that endpoint. I am unsure of what happens in this case; I assume the request will timeout or fail and report an error, which will exit the task and result in a similar failure mode as #2727.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with sled-agent's instance_state_monitor task and trace what happens when the propolis instance_state_monitor endpoint stops responding after a propolis-server panic. Compare the behavior with the failure mode in #2727, then establish tests showing that a panicked server does not leave the instance reported as "Running".
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100