oxidecomputer / oxidecomputer/omicron
Providing an expired cert at RSS time results in an infinite (but doomed) retry loop
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
wicketd does not (and cannot) check the expiration date of the certificate(s) provided as part of the RSS process (since NTP sync hasn't happened yet). If an operator provides an expired-but-otherwise-valid certificate, rack initialization will proceed normally until Nexus attempts to create the recovery silo; that will fail with an error like this:
20:41:04.878Z INFO 1efd6c80-32a9-4edf-8bff-b2b3a4e2e06b (dropshot_internal): request completed
error_message_external = Internal Server Error
error_message_internal = failed to create recovery Silo: Invalid Value: certificate, Certificate exists, but is expired
file = /home/john/.cargo/git/checkouts/dropshot-a4a923d29dccc492/fa728d0/dropshot/src/server.rs:841
latency_us = 679912
local_addr = [fd00:1122:3344:103::5]:12221
method = PUT
remote_addr = [fd00:1122:3344:102::1]:47185
req_id = 0c480403-87d2-4b0f-bdd8-95563b887152
response_code = 500
uri = /racks/9969dd08-23b9-45c1-b05d-afddb4eb2334/initialization-complete
but sled-agent will continue indefinitely trying to instruct Nexus that initialization is complete.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the initialization-complete request from sled-agent to Nexus, using the PUT /racks/.../initialization-complete endpoint and the recovery Silo error described in the issue as the failure boundary. Identify the retry path responsible for continuing after the expired-certificate failure. Done should prevent indefinite retries for this case and cover the behavior with a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100