oxidecomputer / oxidecomputer/crucible

Downstairs reports: database disk image is malformed

Open
#485 15 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

FCS mvp
Dominant language
Rust
Stars
260
Forks
34
Avg merge
2d 1h
Merged PRs (30d)
8

Description

Core(s) generated while omicron was running with 187 disks and one instance. The instance had two disks attached.

Many core files were generated from omicron, with the 2nd oldest being this one from inside crucible.

Core file on catacomb at:
/data/staff/core/gimlet-sn21/20221015/core.oxz_propolis-server_1b5ebd28-3a8f-4013-8927-99c200dbd501.propolis-server.24772

$ mdb core.oxz_propolis-server_1b5ebd28-3a8f-4013-8927-99c200dbd501.propolis-server.24772 
Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]
> $C ! demangle
fffff5ffd07fe470 libc.so.1`_lwp_kill+0xa()
fffff5ffd07fe4a0 libc.so.1`raise+0x1e(6)
fffff5ffd07fe4f0 libc.so.1`abort+0x58()
fffff5ffd07fe500 panic_abort::__rust_start_panic::abort::h5e4ff9722552943b+9()
fffff5ffd07fe510 __rust_start_panic+9()
fffff5ffd07fe5c0 rust_panic+0x1c()
fffff5ffd07fe670 std::panicking::rust_panic_with_hook::ha80c5cb4a11249f7+0x1fe()
fffff5ffd07fe6c0 std::panicking::begin_panic_handler::{{closure}}::h1c465a251fe8dc5f+0xc5()
fffff5ffd07fe6f0 std::sys_common::backtrace::__rust_end_short_backtrace::h73c8afa9397a6a09+0x20()
fffff5ffd07fe730 rust_begin_unwind+0x71()
fffff5ffd07fe770 core::panicking::panic_fmt::h476086f4291cd29f+0x33()
fffff5ffd07fe850 core::panicking::assert_failed_inner::h1c6c74356de69ab3+0xca()
fffff5ffd07fe8b0 core::panicking::assert_failed::ha8afc05e959f3059+0x46()
fffff5ffd07feb00 crucible::Downstairs::retire_check::h3b5d34ae9aea2742+0x4dd()
fffff5ffd07fed90 tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hf1fc0701748dd1a8+0xaa4()
fffff5ffd07ffb10 tokio::runtime::task::harness::Harness<T,S>::poll::ha036dcbe5e688841+0x58()
fffff5ffd07ffbc0 std::thread::local::LocalKey<T>::with::hfaf80dd10aedc955+0x1c8()
fffff5ffd07ffc40 tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h07f63c15f77994f4+0x21f()
fffff5ffd07ffd00 tokio::runtime::scheduler::multi_thread::worker::Context::run::ha897fdb5c3963887+0x72e()
fffff5ffd07ffd50 tokio::runtime::scheduler::multi_thread::worker::run::h2ec47ea0d8bd33fe+0x75()
fffff5ffd07ffd90 tokio::runtime::task::core::CoreStage<T>::poll::h0b439faa5b8bdc5a+0x3f()
fffff5ffd07ffe10 tokio::runtime::task::harness::Harness<T,S>::poll::hbf631161bda3ca94+0x49()
fffff5ffd07ffec0 tokio::runtime::blocking::pool::Inner::run::h24bd9cdb22619ff2+0xbf()
fffff5ffd07fff10 std::sys_common::backtrace::__rust_begin_short_backtrace::hee5d124e97635faa+0x109()
fffff5ffd07fff60 core::ops::function::FnOnce::call_once{{vtable.shim}}::h2fcf5432bb9edc09+0x78()
fffff5ffd07fffb0 std::sys::unix::thread::Thread::new::thread_start::h87fc76126371d726+0x27()
fffff5ffd07fffe0 libc.so.1`_thrp_setup+0x6c(fffff5ffee75a240)
fffff5ffd07ffff0 libc.so.1`_lwp_start()

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the reported core dump and the crucible::Downstairs::retire_check stack frame, using the core file path in the issue. Determine why the malformed database disk image reaches this assertion and reproduce the failure; done means the crash is prevented and the malformed-image case is covered by an appropriate regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.