oxidecomputer / oxidecomputer/omicron

Test flake: silent failure to initialize cockroachdb?

Open
#3,889 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

database Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

https://github.com/oxidecomputer/omicron/pull/3887/checks?check_run_id=15946171931 failed; the immediate symptom in the deploy job a timeout after trying to log in for 10 minutes:

673	2023-08-16T15:51:53.331Z	2023-08-16 15:51:52.586682700 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
674	2023-08-16T15:51:54.332Z	2023-08-16 15:51:53.587485600 UTC: attempting to log into API
675	2023-08-16T15:52:09.366Z	2023-08-16 15:52:08.621092329 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
676	2023-08-16T15:52:10.368Z	2023-08-16 15:52:09.622786274 UTC: attempting to log into API
677	2023-08-16T15:52:25.402Z	2023-08-16 15:52:24.656701751 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
678	2023-08-16T15:52:26.403Z	2023-08-16 15:52:25.657578343 UTC: attempting to log into API
679	2023-08-16T15:52:41.436Z	2023-08-16 15:52:40.691037367 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
680	2023-08-16T15:52:42.439Z	Error: logging in
681	2023-08-16T15:52:42.464Z	
682	2023-08-16T15:52:42.489Z	Caused by:
683	2023-08-16T15:52:43.040Z	    timed out after 609.325828065s

All three nexus logs have many warning that seem to indicate the database wasn't set up correctly, I think?

WARN	nexus: Cannot look up rack: Object (of type ById(24ed7902-3649-4e0e-8635-cd082ae9b0c0)) not found: rack
    file = nexus/src/app/rack.rs:555

but the sled-agent logs do not report any issues with initial CRDB setup:

328	2023-08-16T15:43:19.640Z	INFO	SledAgent (ServiceManager): Formatting CRDB
    file = sled-agent/src/services.rs:2248
...
330	2023-08-16T15:43:22.432Z	INFO	SledAgent (ServiceManager): Formatting CRDB - Completed
    file = sled-agent/src/services.rs:2269

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the failed deploy check from PR 3887 and compare the nexus warnings at nexus/src/app/rack.rs:555 with CRDB formatting in sled-agent/src/services.rs:2248 and :2269. Reproduce the deploy-job login timeout and trace whether initialization is incomplete; done means identifying the failure cause and adding a reliable regression check or fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.