oxidecomputer / oxidecomputer/omicron
Test flake: silent failure to initialize cockroachdb?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
https://github.com/oxidecomputer/omicron/pull/3887/checks?check_run_id=15946171931 failed; the immediate symptom in the deploy job a timeout after trying to log in for 10 minutes:
673 2023-08-16T15:51:53.331Z 2023-08-16 15:51:52.586682700 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
674 2023-08-16T15:51:54.332Z 2023-08-16 15:51:53.587485600 UTC: attempting to log into API
675 2023-08-16T15:52:09.366Z 2023-08-16 15:52:08.621092329 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
676 2023-08-16T15:52:10.368Z 2023-08-16 15:52:09.622786274 UTC: attempting to log into API
677 2023-08-16T15:52:25.402Z 2023-08-16 15:52:24.656701751 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
678 2023-08-16T15:52:26.403Z 2023-08-16 15:52:25.657578343 UTC: attempting to log into API
679 2023-08-16T15:52:41.436Z 2023-08-16 15:52:40.691037367 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
680 2023-08-16T15:52:42.439Z Error: logging in
681 2023-08-16T15:52:42.464Z
682 2023-08-16T15:52:42.489Z Caused by:
683 2023-08-16T15:52:43.040Z timed out after 609.325828065s
All three nexus logs have many warning that seem to indicate the database wasn't set up correctly, I think?
WARN nexus: Cannot look up rack: Object (of type ById(24ed7902-3649-4e0e-8635-cd082ae9b0c0)) not found: rack
file = nexus/src/app/rack.rs:555
but the sled-agent logs do not report any issues with initial CRDB setup:
328 2023-08-16T15:43:19.640Z INFO SledAgent (ServiceManager): Formatting CRDB
file = sled-agent/src/services.rs:2248
...
330 2023-08-16T15:43:22.432Z INFO SledAgent (ServiceManager): Formatting CRDB - Completed
file = sled-agent/src/services.rs:2269
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the failed deploy check from PR 3887 and compare the nexus warnings at nexus/src/app/rack.rs:555 with CRDB formatting in sled-agent/src/services.rs:2248 and :2269. Reproduce the deploy-job login timeout and trace whether initialization is incomplete; done means identifying the failure cause and adding a reliable regression check or fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100