oxidecomputer / oxidecomputer/omicron
Nexus panicked on no CA certs loaded in some of the background tasks
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
On rack2, I found a nexus core file and the log search indicates that it panicked due to the lack of a CA cert:
thread 'tokio-rt-worker' (41) panicked at clients/dns-service-client/src/lib.rs:5:1:
called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Builder, source: General("No CA certificates were loaded from the system") }
{"msg":"authorize result","v":0,"name":"nexus","level":20,"time":"2026-07-19T06:15:56.133544466Z","hostname":"oxz_nexus_6a11df2d-df48-415e-a5d4-bd591d593c96","pid":20384,"background_task":"aban
doned_vmm_reaper","component":"BackgroundTasks","component":"nexus","component":"ServerContext","name":"6a11df2d-df48-415e-a5d4-bd591d593c96","result":"Ok(())","resource":"Database","action":"Q
uery","actor":"Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002 (built_in_user), .. })"}
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[ Jul 19 06:16:06 Stopping because all processes in service exited. ]
[ Jul 19 06:16:06 Executing stop method (:kill). ]
[ Jul 19 06:16:06 Executing start method ("ctrun -l child -o noorphan,regent /opt/oxide/omicron-nexus/bin/nexus /var/svc/manifest/site/nexus/config.toml &"). ]
[ Jul 19 06:16:06 Method "start" exited with status 0. ]
I am pretty sure we have certs at the time (we aren't crossing the expiration date right at that moment).
The rack had 21.0.0-0.ci+git3e5bafe8ad9 when nexus panicked.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at clients/dns-service-client/src/lib.rs:5:1 and inspect the unwrap that panics when reqwest reports that no system CA certificates were loaded. Use the rack logs from version 21.0.0-0.ci+git3e5bafe8ad9 to trace why certificates were unavailable in the background task; done means Nexus no longer panics in this condition and the failure is handled appropriately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100