oxidecomputer / oxidecomputer/omicron

Zpool not marked as deleted in database after disk expungement

Open
#7,962 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Add/expunge sled expunge
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

I was trying out the new omdb db crucible-dataset command on rack2 but kept hitting this error:

root@oxz_switch0:~# omdb db zpool list
Error: zpool 3af01cc4-1f16-47d9-a489-abafcb91c2db not found in inventory!

This zpool corresponds to an expunged physical disk:

root@[fd00:1122:3344:105::3]:32221/omicron> select * from zpool where id = '3af01cc4-1f16-47d9-a489-abafcb91c2db';
                   id                  |         time_created          |         time_modified         | time_deleted | rcgen |               sled_id                |           physical_disk_id           | control_plane_storage_buffer
---------------------------------------+-------------------------------+-------------------------------+--------------+-------+--------------------------------------+--------------------------------------+-------------------------------
  3af01cc4-1f16-47d9-a489-abafcb91c2db | 2023-08-30 18:57:52.826594+00 | 2024-03-27 21:18:20.842573+00 | NULL         |     4 | 0c7011f7-a4bf-4daf-90cc-1c2410103300 | 0bffef3c-56b1-4b01-b897-161bbf766bea |                 268435456000
(1 row)
root@[fd00:1122:3344:105::3]:32221/omicron> select * from physical_disk where id = '0bffef3c-56b1-4b01-b897-161bbf766bea';
                   id                  |         time_created          |         time_modified         | time_deleted | rcgen | vendor |  serial  |      model      | variant |               sled_id                | disk_policy |   disk_state
---------------------------------------+-------------------------------+-------------------------------+--------------+-------+--------+----------+-----------------+---------+--------------------------------------+-------------+-----------------
  0bffef3c-56b1-4b01-b897-161bbf766bea | 2023-08-30 18:57:23.655377+00 | 2024-03-27 21:18:20.906665+00 | NULL         |     1 | 1b96   | A084A5EE | WUS4C6432DSP3X3 | u2      | 0c7011f7-a4bf-4daf-90cc-1c2410103300 | expunged    | decommissioned
(1 row)

We can probably make omdb more forgiving but the proper fix may be cleaning up the orphan record for a zpool that is permanently gone from the sled.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the omdb db zpool list failure and inspecting the reported zpool and physical_disk records, including their deletion and expungement fields. Determine the intended handling for a zpool tied to an expunged disk; done means the orphaned record no longer causes the inventory error and the relevant database command behaves consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.