oxidecomputer / oxidecomputer/omicron

RSS stuck for single-node non-gimlet deployment due to missing zpool?

Open
#4,202 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

development
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

I don't quite understand what's going on here, but I want to teardown this deployment and try standing up omicron again so I'm not blocked, so I'm dumping some state for now.

I deployed omicron on dunkin, a lab machine. The stack never quite came up, with this message repeatedly showing up from sled agent:

    file = sled-agent/src/rack_setup/service.rs:627
18:07:10.200Z INFO SledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "09f78537-e5f6-4452-82d2-280e443e9e69", "content-length": "174", "date": "Wed, 04 Oct 2023 18:07:09 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: zpool with id \"616b26df-e62a-4c68-b506-f4a923d8aaf7\"", request_id: "09f78537-e5f6-4452-82d2-280e443e9e69" }

That zpool exists on this system:

$ zpool status oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
  pool: oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
 state: ONLINE
  scan: none requested
config:

        NAME                                                                           STATE     READ WRITE CKSUM
        oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7                                       ONLINE       0     0     0
          /home/jordan/omicron/tools/../oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev  ONLINE       0     0     0

errors: No known data errors

I looked for evidence of this request on the 3 nexuses in my deployment, and couldn't find a matching request (I looked for request ID and also just grepped for that zpool uuid). That log line comes from here:

https://github.com/oxidecomputer/omicron/blob/a9104a0786f437e38d4945dd77470bc8fd4f974c/sled-agent/src/rack_setup/service.rs#L608-L635

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at sled-agent/src/rack_setup/service.rs around lines 608-635 and trace the RSS handoff to Nexus. Reproduce or inspect the single-node non-gimlet deployment state, comparing the zpool UUID reported by sled agent with the zpool visible to Nexus. Done should identify why the existing zpool is reported as missing and document a verified resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.