oxidecomputer / oxidecomputer/crucible
Secret panic hidden in `test_volume_replace_downstairs_then_takeover`
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 260
- Forks
- 34
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 8
Description
Looking at state transitions in test_volume_replace_downstairs_then_takeover, I see the following:
INFO [0] New New New ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [1] WaitActive New New ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [2] WaitActive WaitActive New ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [0] WaitActive WaitActive WaitActive ds_transition to WaitQuorum, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [1] WaitQuorum WaitActive WaitActive ds_transition to WaitQuorum, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [2] WaitQuorum WaitQuorum WaitActive ds_transition to WaitQuorum, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [0] Active Active Active ds_transition to Replacing, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [0] New New New ds_transition to WaitActive, session_id: 638e7625-29e5-4a1f-b779-45f5fdbf1f17
INFO [1] WaitActive New New ds_transition to WaitActive, session_id: 638e7625-29e5-4a1f-b779-45f5fdbf1f17
INFO [2] WaitActive WaitActive New ds_transition to WaitActive, session_id: 638e7625-29e5-4a1f-b779-45f5fdbf1f17
INFO [1] Replaced Active Active ds_transition to Disabled, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [2] Replaced Disconnected Active ds_transition to Disabled, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [0] WaitActive WaitActive WaitActive ds_transition to WaitQuorum, session_id: 638e7625-29e5-4a1f-b779-45f5fdbf1f17
INFO [1] WaitQuorum WaitActive WaitActive ds_transition to WaitQuorum, session_id: 638e7625-29e5-4a1f-b779-45f5fdbf1f17
INFO [2] WaitQuorum WaitQuorum WaitActive ds_transition to WaitQuorum, session_id: 638e7625-29e5-4a1f-b779-45f5fdbf1f17
INFO [0] Replaced Disconnected Disconnected ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [1] Replaced Disconnected Disconnected ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
INFO [2] Replaced WaitActive Disconnected ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
It turns out that going from Replaced to WaitActive is illegal, and we in fact see a panic in the logs:
Nov 03 17:51:48.435 INFO [0] 77d7dfc7-e462-496e-9be6-876f133cbcfe (c9f8ab22-b719-48a1-be58-eb7e8a733e95) Replaced Disconnected Disconnected ds_transition to WaitActive, session_id: c9f8ab22-b719-48a1-be58-eb7e8a733e95
thread 'test::integration_test_volume_replace_downstairs_then_takeover' panicked at '[0] 77d7dfc7-e462-496e-9be6-876f133cbcfe Initializing Negotiation failed, Replaced -> WaitActive', upstairs/src/lib.rs:6334:21
This happens to work, because the tokio::sync::Mutex<Downstairs> just silently unlocks on panics (and kills the task, which is unused in this test). However, it's causing difficulties in my experiments with std::sync::Mutex, because that flavor of mutex is poisoned on panic.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the integration test test_volume_replace_downstairs_then_takeover and inspect the transition handling around upstairs/src/lib.rs:6334. Reproduce the test and trace the Replaced to WaitActive transition; done should address the reported panic rather than relying on Tokio mutex behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100