oxidecomputer / oxidecomputer/omicron

Nexus busted after its initial startup raced with another Nexus populating "system" VpcRouter

Open
#5,980 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Here's the CI failure:
https://github.com/oxidecomputer/omicron/pull/5962/checks?check_run_id=26826135569

This PR is non-trivial and it's conceivable that my changes introduced this but I don't yet see how. The helios-deploy job failed in the usual way (timing out after 600s trying to log in):

2024-06-28 22:19:52.051692811 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: error:0A000419:SSL routines:ssl3_read_bytes:tlsv1 alert access denied:ssl/record/rec_layer_s3.c:1605:SSL alert number 49
Error: logging in

Caused by:
    timed out after 600.598279588s

That TLS error can reflect that there are no certificates, and so a problem with the "external endpoints" task. I went to the Nexus logs to look for errors and found that one Nexus failed some of the first-time-setup steps:

2024-06-28T22:10:18.588Z	ERRO	nexus (DataLoader): gave up trying to populate built-in PopulateBuiltinVpcs
error_message = InternalError { internal_message: "Unknown diesel error creating VpcRouter called \\"system\\": Record not found" }
file = nexus/src/populate.rs:126
2024-06-28T22:10:18.588Z	ERRO	nexus: populate failed
file = nexus/src/app/mod.rs:528
2024-06-28T22:10:18.588Z	ERRO	nexus: saga request channel closed!
file = nexus/src/app/mod.rs:544

One possibility here is that before my PR, we would still have run the background tasks that eventually configure TLS, whereas now we won't. But at best the old code would have been papering over this problem.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with nexus/src/populate.rs:126 and nexus/src/app/mod.rs:528-544, then inspect the linked CI failure and Nexus logs. Determine how concurrent initial population of the "system" VpcRouter affects the setup sequence and TLS-related background tasks. Done means the startup path reliably completes without the login timeout or population failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.