oxidecomputer / oxidecomputer/omicron
sled-agent panics when out of disk space for slog
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
A core was recovered from the field with this stack:
> $C
fffff5ffcdbff9b0 libc.so.1`_lwp_kill+0xa()
fffff5ffcdbff9e0 libc.so.1`raise+0x22(6)
fffff5ffcdbffa30 libc.so.1`abort+0x58()
fffff5ffcdbffa40 _ZN11panic_abort18__rust_start_panic5abort17h0cfb9e9813446f8bE+9()
fffff5ffcdbffa50 __rust_start_panic+9()
fffff5ffcdbffaf0 rust_panic+0x10()
fffff5ffcdbffba0 _ZN3std9panicking20rust_panic_with_hook17ha18432c291108e47E+0x2a2()
fffff5ffcdbffbf0 _ZN3std9panicking19begin_panic_handler28_$u7b$$u7b$closure$u7d$$u7d$17h4645986a55c3ef2bE+0xc6()
fffff5ffcdbffc00 _ZN3std10sys_common9backtrace26__rust_end_short_backtrace17h6107fb71e173f9a5E+9()
fffff5ffcdbffc40 rust_begin_unwind+0x71()
fffff5ffcdbffc80 _ZN4core9panicking9panic_fmt17h863016252fdb1147E+0x33()
fffff5ffcdbffcf0 _ZN51_$LT$slog..Fuse$LT$D$GT$$u20$as$u20$slog..Drain$GT$3log17h9d63047a523045c3E+0x96()
fffff5ffcdbffe80 _ZN3std9panicking3try17h42cacb52dc536ef0E+0x163()
fffff5ffcdbffec0 _ZN3std10sys_common9backtrace28__rust_begin_short_backtrace17h490aebce9851cb53E+0x25()
fffff5ffcdbfff60 _ZN4core3ops8function6FnOnce40call_once$u7b$$u7b$vtable.shim$u7d$$u7d$17hc6b68a1c69ad316bE+0x99()
fffff5ffcdbfffb0 _ZN3std3sys4unix6thread6Thread3new12thread_start17h0604336de98f7b8bE+0x29()
fffff5ffcdbfffe0 libc.so.1`_thrp_setup+0x77(fffff5ffedd60240)
fffff5ffcdbffff0 libc.so.1`_lwp_start()
The root filesystem on this machine had no free space available for reasons not related to sled-agent. This appears substantially identical in nature to #4352 but affects a different entity. Of possible interest is that fact that if we were to implement some kind of "try to free up storage space for logs" facility, it would almost certainly be sled-agent itself responsible for doing so. Regardless, this isn't a reason to panic.
Core file is catacomb:/data/staff/core/customer-support/52/pool/ext/2081704d-aed2-4676-92ce-7f8a576d66ad/crypt/debug/core.global.sled-agent.22023.1697600174.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the sled-agent logging path shown in the stack, especially slog::Fuse, and compare the behavior with issue #4352. Determine how disk-full logging errors are handled; done means sled-agent does not panic when the root filesystem has no free space, with regression coverage for this condition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100