oxidecomputer / oxidecomputer/omicron

Providing an expired cert at RSS time results in an infinite (but doomed) retry loop

Open
#4,205 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

wicketd does not (and cannot) check the expiration date of the certificate(s) provided as part of the RSS process (since NTP sync hasn't happened yet). If an operator provides an expired-but-otherwise-valid certificate, rack initialization will proceed normally until Nexus attempts to create the recovery silo; that will fail with an error like this:

20:41:04.878Z INFO 1efd6c80-32a9-4edf-8bff-b2b3a4e2e06b (dropshot_internal): request completed
    error_message_external = Internal Server Error
    error_message_internal = failed to create recovery Silo: Invalid Value: certificate, Certificate exists, but is expired
    file = /home/john/.cargo/git/checkouts/dropshot-a4a923d29dccc492/fa728d0/dropshot/src/server.rs:841
    latency_us = 679912
    local_addr = [fd00:1122:3344:103::5]:12221
    method = PUT
    remote_addr = [fd00:1122:3344:102::1]:47185
    req_id = 0c480403-87d2-4b0f-bdd8-95563b887152
    response_code = 500
    uri = /racks/9969dd08-23b9-45c1-b05d-afddb4eb2334/initialization-complete

but sled-agent will continue indefinitely trying to instruct Nexus that initialization is complete.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the initialization-complete request from sled-agent to Nexus, using the PUT /racks/.../initialization-complete endpoint and the recovery Silo error described in the issue as the failure boundary. Identify the retry path responsible for continuing after the expired-certificate failure. Done should prevent indefinite retries for this case and cover the behavior with a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.