oxidecomputer / oxidecomputer/omicron

omicron-stress can trigger sled-agent panic: "should have an active Propolis zone by now"

Open
#10,369 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Running omicron-stress with the patch

diff --git a/src/actor/instance.rs b/src/actor/instance.rs
index 7c14b94..cd0593b 100644
--- a/src/actor/instance.rs
+++ b/src/actor/instance.rs
@@ -120,7 +120,7 @@ impl InstanceActor {
             ncpus: oxide::types::InstanceCpuCount(1),
             network_interfaces:
                 oxide::types::InstanceNetworkInterfaceAttachment::None,
-            start: true,
+            start: false,
             user_data: String::new(),
             ssh_public_keys: None,
         };

and the invocation:

exec cargo run --release -- \
    --num-test-instances 8 \
    --threads-per-instance 8 \
    --num-test-disks 0 \
    --threads-per-disk 0 \
    --num-test-snapshots 0 \
    --threads-per-snapshot 0 \
    --server-errors-fatal

In the Canada region, it seems I can reliably get sled-agent to panic after a few invocations:

thread 'tokio-rt-worker' (30) panicked at sled-agent/src/instance.rs:2286:9:
should have an active Propolis zone by now
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the shown cargo run --release invocation and the start: false change in src/actor/instance.rs. Then inspect sled-agent/src/instance.rs around line 2286 and trace the instance lifecycle leading to the panic. Done means repeated omicron-stress runs no longer trigger the reported sled-agent panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.