oxidecomputer / oxidecomputer/omicron

sled-agent crash potentially related to a pfile process

Open
#8,697 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Found a core file on sled 23 of rack2:

> ::status
debugging core file of sled-agent (64-bit) from BRM42220016
file: /opt/oxide/sled-agent/sled-agent
initial argv: /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml
threading model: raw lwps
status: process terminated by SIGSEGV (Segmentation Fault), addr=0

BRM42220016 # mdb /pool/ext/9dfe424f-cba6-4bfb-a3dd-e8bd7fdea57d/crypt/debug/core.global.sled-agent.654.1753458407
mdb: agent lwp detected; forcing lwp thread model (use ::tmodel to change)
Loading modules: [ libumem.so.1 libnvpair.so.1 libc.so.1 ld.so.1 ]
> $C ! demangle
fffff7ffed9ff850 libc.so.1`cond_wait_queue+0x55(5477a20, 5477a30, 0)
fffff7ffed9ff8a0 libc.so.1`__cond_wait+0xba(5477a20, 5477a30)
fffff7ffed9ff8d0 libc.so.1`cond_wait+0x2e(5477a20, 5477a30)
fffff7ffed9ff8f0 libc.so.1`pthread_cond_wait+0x15(5477a20, 5477a30)
fffff7ffed9ffa10 parking_lot::condvar::Condvar::wait_until_internal::h245225b95baa609c+0x2ab()
fffff7ffed9ffa90 tokio::runtime::scheduler::multi_thread::park::Parker::park::ha39d53c7fdd2d9ee+0x100()
fffff7ffed9ffb20 tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout::hb6a032352bccdf03+0x79()
fffff7ffed9ffbb0 tokio::runtime::scheduler::multi_thread::worker::Context::run::ha6137785420c341f+0x106d()
fffff7ffed9ffc10 tokio::runtime::context::scoped::Scoped<T>::set::h404d11d03fae2798+0x2a()
fffff7ffed9ffcd0 tokio::runtime::context::runtime::enter_runtime::hbea92029ca951a9a+0xb1()
fffff7ffed9ffd10 tokio::runtime::scheduler::multi_thread::worker::run::h85e66819c406cdd0+0xa0()
fffff7ffed9ffd70 tokio::runtime::task::core::Core<T,S>::poll::hd8de6c001086b642+0x70()
fffff7ffed9ffdc0 tokio::runtime::task::harness::Harness<T,S>::poll::hd41585bd4a0d94e4+0x8a()
fffff7ffed9ffe70 tokio::runtime::blocking::pool::Inner::run::hfccadbb87964bbce+0xe4()
fffff7ffed9ffec0 std::sys::backtrace::__rust_begin_short_backtrace::h0d8835be7d5c7d5b+0x72()
fffff7ffed9fff60 core::ops::function::FnOnce::call_once{{vtable.shim}}::h2c58797cb5db6f94+0x97()
fffff7ffed9fffb0 std::sys::pal::unix::thread::Thread::new::thread_start::hb8b719afb1b31961+0x2b()
fffff7ffed9fffe0 libc.so.1`_thrp_setup+0x77(fffff7ffeeef0a40)
fffff7ffed9ffff0 libc.so.1`_lwp_start()

There was nothing notable in the sled-agent log during the crash.

We know that a support bundle happens to be running at the time which might have invoked pfile. This is the bundle that was kicked off shortly before the crash:

{
    "id": "0a7a86c8-6e03-4a11-a89b-12dcb0464c1b",
    "reason_for_creation": "Created by internal API",
    "state": "active",
    "time_created": "2025-07-25T15:46:18.620587Z"
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the sled-agent core at /pool/ext/9dfe424f-cba6-4bfb-a3dd-e8bd7fdea57d/crypt/debug/core.global.sled-agent.654.1753458407 and inspect the mdb stack, then review the sled-agent logs and the support-bundle timing. Determine whether the concurrent pfile activity is related to the SIGSEGV and document or address the confirmed cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.