oxidecomputer / oxidecomputer/omicron
sled-agent crash potentially related to a pfile process
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Found a core file on sled 23 of rack2:
> ::status
debugging core file of sled-agent (64-bit) from BRM42220016
file: /opt/oxide/sled-agent/sled-agent
initial argv: /opt/oxide/sled-agent/sled-agent run /opt/oxide/sled-agent/pkg/config.toml
threading model: raw lwps
status: process terminated by SIGSEGV (Segmentation Fault), addr=0
BRM42220016 # mdb /pool/ext/9dfe424f-cba6-4bfb-a3dd-e8bd7fdea57d/crypt/debug/core.global.sled-agent.654.1753458407
mdb: agent lwp detected; forcing lwp thread model (use ::tmodel to change)
Loading modules: [ libumem.so.1 libnvpair.so.1 libc.so.1 ld.so.1 ]
> $C ! demangle
fffff7ffed9ff850 libc.so.1`cond_wait_queue+0x55(5477a20, 5477a30, 0)
fffff7ffed9ff8a0 libc.so.1`__cond_wait+0xba(5477a20, 5477a30)
fffff7ffed9ff8d0 libc.so.1`cond_wait+0x2e(5477a20, 5477a30)
fffff7ffed9ff8f0 libc.so.1`pthread_cond_wait+0x15(5477a20, 5477a30)
fffff7ffed9ffa10 parking_lot::condvar::Condvar::wait_until_internal::h245225b95baa609c+0x2ab()
fffff7ffed9ffa90 tokio::runtime::scheduler::multi_thread::park::Parker::park::ha39d53c7fdd2d9ee+0x100()
fffff7ffed9ffb20 tokio::runtime::scheduler::multi_thread::worker::Context::park_timeout::hb6a032352bccdf03+0x79()
fffff7ffed9ffbb0 tokio::runtime::scheduler::multi_thread::worker::Context::run::ha6137785420c341f+0x106d()
fffff7ffed9ffc10 tokio::runtime::context::scoped::Scoped<T>::set::h404d11d03fae2798+0x2a()
fffff7ffed9ffcd0 tokio::runtime::context::runtime::enter_runtime::hbea92029ca951a9a+0xb1()
fffff7ffed9ffd10 tokio::runtime::scheduler::multi_thread::worker::run::h85e66819c406cdd0+0xa0()
fffff7ffed9ffd70 tokio::runtime::task::core::Core<T,S>::poll::hd8de6c001086b642+0x70()
fffff7ffed9ffdc0 tokio::runtime::task::harness::Harness<T,S>::poll::hd41585bd4a0d94e4+0x8a()
fffff7ffed9ffe70 tokio::runtime::blocking::pool::Inner::run::hfccadbb87964bbce+0xe4()
fffff7ffed9ffec0 std::sys::backtrace::__rust_begin_short_backtrace::h0d8835be7d5c7d5b+0x72()
fffff7ffed9fff60 core::ops::function::FnOnce::call_once{{vtable.shim}}::h2c58797cb5db6f94+0x97()
fffff7ffed9fffb0 std::sys::pal::unix::thread::Thread::new::thread_start::hb8b719afb1b31961+0x2b()
fffff7ffed9fffe0 libc.so.1`_thrp_setup+0x77(fffff7ffeeef0a40)
fffff7ffed9ffff0 libc.so.1`_lwp_start()
There was nothing notable in the sled-agent log during the crash.
We know that a support bundle happens to be running at the time which might have invoked pfile. This is the bundle that was kicked off shortly before the crash:
{
"id": "0a7a86c8-6e03-4a11-a89b-12dcb0464c1b",
"reason_for_creation": "Created by internal API",
"state": "active",
"time_created": "2025-07-25T15:46:18.620587Z"
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the sled-agent core at /pool/ext/9dfe424f-cba6-4bfb-a3dd-e8bd7fdea57d/crypt/debug/core.global.sled-agent.654.1753458407 and inspect the mdb stack, then review the sled-agent logs and the support-bundle timing. Determine whether the concurrent pfile activity is related to the SIGSEGV and document or address the confirmed cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100