oxidecomputer / oxidecomputer/omicron

Transient failures in instance state monitor cause instances to be lost at sea

Open
#2,727 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Sled Agent
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Sled agent spawns a state monitor task per instance that calls Propolis's instance_state_monitor endpoint, watches for VM state transitions, processes them, and relays any resulting instance state changes to Nexus. Any error in Instance::monitor_state_task or any of its callees bubbles up through the task and causes it to exit. After this, nothing monitors Propolis for subsequent state changes or notifies Nexus if they occur.

The DNS resolution error in #2726 brought this to my immediate attention, but any failure in the monitoring task will do the trick; another not-too-esoteric one is a Propolis panic that causes the call to instance_state_monitor to fail.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the Sled agent at Instance::monitor_state_task and trace its callees to the instance_state_monitor endpoint, including how errors exit the task and how changes are relayed to Nexus. Reproduce or test a transient monitor failure, then verify that later Propolis state transitions are still observed and reported to Nexus.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.