oxidecomputer / oxidecomputer/omicron
RSS stuck for single-node non-gimlet deployment due to missing zpool?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
I don't quite understand what's going on here, but I want to teardown this deployment and try standing up omicron again so I'm not blocked, so I'm dumping some state for now.
I deployed omicron on dunkin, a lab machine. The stack never quite came up, with this message repeatedly showing up from sled agent:
file = sled-agent/src/rack_setup/service.rs:627
18:07:10.200Z INFO SledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "09f78537-e5f6-4452-82d2-280e443e9e69", "content-length": "174", "date": "Wed, 04 Oct 2023 18:07:09 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: zpool with id \"616b26df-e62a-4c68-b506-f4a923d8aaf7\"", request_id: "09f78537-e5f6-4452-82d2-280e443e9e69" }
That zpool exists on this system:
$ zpool status oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
pool: oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 ONLINE 0 0 0
/home/jordan/omicron/tools/../oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ONLINE 0 0 0
errors: No known data errors
I looked for evidence of this request on the 3 nexuses in my deployment, and couldn't find a matching request (I looked for request ID and also just grepped for that zpool uuid). That log line comes from here:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at sled-agent/src/rack_setup/service.rs around lines 608-635 and trace the RSS handoff to Nexus. Reproduce or inspect the single-node non-gimlet deployment state, comparing the zpool UUID reported by sled agent with the zpool visible to Nexus. Done should identify why the existing zpool is reported as missing and document a verified resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100