oxidecomputer / oxidecomputer/crucible

Secret panic hidden in `test_volume_replace_downstairs_then_takeover`

Open
#1,017 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
260
Forks
34
Avg merge
2d 1h
Merged PRs (30d)
8

Description

Looking at state transitions in test_volume_replace_downstairs_then_takeover, I see the following:

INFO [0]  New New New ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [1]  WaitActive New New ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [2]  WaitActive WaitActive New ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [0]  WaitActive WaitActive WaitActive ds_transition to WaitQuorum, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [1]  WaitQuorum WaitActive WaitActive ds_transition to WaitQuorum, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [2]  WaitQuorum WaitQuorum WaitActive ds_transition to WaitQuorum, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95

INFO [0]  Active Active Active ds_transition to Replacing, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
    INFO [0]  New New New ds_transition to WaitActive, session_id: 638e7625-29e5-4a1f-b779-45f5fdbf1f17
    INFO [1]  WaitActive New New ds_transition to WaitActive, session_id: 638e7625-29e5-4a1f-b779-45f5fdbf1f17
    INFO [2]  WaitActive WaitActive New ds_transition to WaitActive, session_id: 638e7625-29e5-4a1f-b779-45f5fdbf1f17

INFO [1]  Replaced Active Active ds_transition to Disabled, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [2]  Replaced Disconnected Active ds_transition to Disabled, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95

    INFO [0]  WaitActive WaitActive WaitActive ds_transition to WaitQuorum, session_id: 638e7625-29e5-4a1f-b779-45f5fdbf1f17
    INFO [1]  WaitQuorum WaitActive WaitActive ds_transition to WaitQuorum, session_id: 638e7625-29e5-4a1f-b779-45f5fdbf1f17
    INFO [2]  WaitQuorum WaitQuorum WaitActive ds_transition to WaitQuorum, session_id: 638e7625-29e5-4a1f-b779-45f5fdbf1f17

INFO [0]  Replaced Disconnected Disconnected ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [1]  Replaced Disconnected Disconnected ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [2]  Replaced WaitActive Disconnected ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95

It turns out that going from Replaced to WaitActive is illegal, and we in fact see a panic in the logs:

Nov 03 17:51:48.435 INFO [0] 77d7dfc7-e462-496e-9be6-876f133cbcfe (c9f8ab22-b719-48a1-be58-eb7e8a733e95) Replaced Disconnected Disconnected ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
thread 'test::integration_test_volume_replace_downstairs_then_takeover' panicked at '[0] 77d7dfc7-e462-496e-9be6-876f133cbcfe Initializing Negotiation failed, Replaced -> WaitActive', upstairs/src/lib.rs:6334:21

This happens to work, because the tokio::sync::Mutex<Downstairs> just silently unlocks on panics (and kills the task, which is unused in this test). However, it's causing difficulties in my experiments with std::sync::Mutex, because that flavor of mutex is poisoned on panic.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the integration test test_volume_replace_downstairs_then_takeover and inspect the transition handling around upstairs/src/lib.rs:6334. Reproduce the test and trace the Replaced to WaitActive transition; done should address the reported panic rather than relying on Tokio mutex behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.