oxidecomputer / oxidecomputer/omicron

sled-agent failed to notice propolis-server panicked

Open
#3,206 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Sled Agent
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

The propolis-server running an instance on rack2 panicked, but the instance state continued to be reported in the console as "Running". The instance_state_monitor task in sled-agent is responsible for observing state changes from propolis, which loops calling the instance_state_monitor endpoint. If propolis-server panics, then propolis will stop returning requests from that endpoint. I am unsure of what happens in this case; I assume the request will timeout or fail and report an error, which will exit the task and result in a similar failure mode as #2727.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with sled-agent's instance_state_monitor task and trace what happens when the propolis instance_state_monitor endpoint stops responding after a propolis-server panic. Compare the behavior with the failure mode in #2727, then establish tests showing that a panicked server does not leave the instance reported as "Running".

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.