oxidecomputer / oxidecomputer/omicron

[sled-agent-sim] Restart leaves broken datasets in Cockroach

Open
#658 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

development
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

This is not blocking us on Console yet. We can work locally just fine, and we don't especially need to do anything with disks on GCP QA right now. Just documenting the issue.

Problem

We are seeing 500s on disk create in the copy of Console and Nexus deployed to GCP. We do not see these running locally.

(http://127.0.0.1:42401/crucible/0/regions): error trying to connect: tcp connect error: Connection refused (os error 111)

Here’s what’s happening:

  1. Sim sled agent spins up 10 Crucible servers and creates 10 corresponding datasets at startup during packer step and registers them with Nexus, which means they get entries in the DB
  2. At GCP instance boot, sled agent mysteriously crashes (taking Crucible servers along with it) and restarts, making a new sled agent with 10 new crucible servers and corresponding dataset entries in the DB
  3. At this point disk allocation fails in Nexus because it tries to use the earlier datasets, which no longer have Crucible servers listening on those ports

Story confirmed by the fact that it works if you make Nexus prefer the most recent datasets when creating a disk.

Expand to see the log of the two sets of 10 crucible server starts 
david_crespo@console-git-bump-api:~$ docker logs sled-agent
Jan 27 19:12:44.093 INFO setting up sled agent server
Jan 27 19:12:44.122 INFO created simulated sled agent, sim_mode: Auto, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:12:44.447 INFO Created Simulated Crucible Server, address: 0.0.0.0:41783, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:12:44.712 INFO Created Simulated Crucible Server, address: 0.0.0.0:39409, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:12:44.785 INFO Created Simulated Crucible Server, address: 0.0.0.0:33869, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:12:44.874 INFO Created Simulated Crucible Server, address: 0.0.0.0:44129, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:12:44.945 INFO Created Simulated Crucible Server, address: 0.0.0.0:41809, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:12:45.017 INFO Created Simulated Crucible Server, address: 0.0.0.0:37955, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:12:45.111 INFO Created Simulated Crucible Server, address: 0.0.0.0:44177, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:12:45.201 INFO Created Simulated Crucible Server, address: 0.0.0.0:35009, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:12:45.262 INFO Created Simulated Crucible Server, address: 0.0.0.0:46475, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:12:45.315 INFO Created Simulated Crucible Server, address: 0.0.0.0:37405, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:12:45.342 INFO sled agent started successfully
Jan 27 19:17:56.390 INFO setting up sled agent server
Jan 27 19:17:56.517 INFO created simulated sled agent, sim_mode: Auto, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:17:56.565 WARN failed to contact nexus, will retry in 127.57591ms, error: error sending request for url (http://0.0.0.0:12221/sled_agents/5223674a-d003-4149-97a9-89330cc1bffd): error trying to connect: tcp connect error: Connection refused (os error 
111)
Caused by:
    0: error trying to connect: tcp connect error: Connection refused (os error 111)
    1: tcp connect error: Connection refused (os error 111)
    2: Connection refused (os error 111)
Jan 27 19:17:56.694 WARN failed to contact nexus, will retry in 548.001243ms, error: error sending request for url (http://0.0.0.0:12221/sled_agents/5223674a-d003-4149-97a9-89330cc1bffd): error trying to connect: tcp connect error: Connection refused (os error
 111)
Caused by:
    0: error trying to connect: tcp connect error: Connection refused (os error 111)
    1: tcp connect error: Connection refused (os error 111)
    2: Connection refused (os error 111)
Jan 27 19:17:57.244 WARN failed to contact nexus, will retry in 1.17709359s, error: error sending request for url (http://0.0.0.0:12221/sled_agents/5223674a-d003-4149-97a9-89330cc1bffd): error trying to connect: tcp connect error: Connection refused (os error 
111)
Caused by:
    0: error trying to connect: tcp connect error: Connection refused (os error 111)
    1: tcp connect error: Connection refused (os error 111)
    2: Connection refused (os error 111)
Jan 27 19:18:08.179 INFO Created Simulated Crucible Server, address: 0.0.0.0:46289, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:18:08.555 INFO Created Simulated Crucible Server, address: 0.0.0.0:33293, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:18:08.640 INFO Created Simulated Crucible Server, address: 0.0.0.0:35507, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:18:08.710 INFO Created Simulated Crucible Server, address: 0.0.0.0:40333, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:18:08.752 INFO Created Simulated Crucible Server, address: 0.0.0.0:39349, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:18:08.792 INFO Created Simulated Crucible Server, address: 0.0.0.0:35055, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:18:08.839 INFO Created Simulated Crucible Server, address: 0.0.0.0:35835, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:18:08.904 INFO Created Simulated Crucible Server, address: 0.0.0.0:33285, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:18:08.948 INFO Created Simulated Crucible Server, address: 0.0.0.0:42807, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:18:09.002 INFO Created Simulated Crucible Server, address: 0.0.0.0:32937, kind: storage, server: 5223674a-d003-4149-97a9-89330cc1bffd, component: SledAgent
Jan 27 19:18:09.026 INFO sled agent started successfully

Solutions

While the proximate cause here is a restart that I think is caused by the Docker-based setup we're using to deploy to GCP, it does show that Nexus doesn't handle it well when Sled Agents and Crucibles go down. I talked about it with @smklein and he already plans for Nexus to handle sled agent failures more robustly, with the ability for Nexus to mark service instances and try to talk to a different one instead. This is discussed in RFD 237.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with sled-agent/src/sim/server.rs and the linked packer/bootstrap setup, then read RFD 237 and trace Nexus handling of sled-agent and dataset service instances. Reproduce the restart scenario and identify the relevant failure-handling entry point. The issue does not define a concrete implementation or test, so done would require agreeing on scope and verifying that disk creation no longer selects unavailable datasets.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, rust
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.