oxidecomputer / oxidecomputer/omicron

Sled agent fails to adopt disk with 4096 LBA and no GPT labels

Open
#11,040 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Sled Agent
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Typically disks arrive from the factory with an LBA of 512 and no GPT label set, and adoption sets the LBA to 4096. Recently we needed to remediate some U.2 drives where the LBA remained 512 post-adoption.

After destroying the zpool, we used nvmeadm format nvme4 1, to set the block size to 4096. However, when sled-agent failed to adopt this disk:

00:04:59.662Z WARN SledAgent (ConfigReconcilerTask): Disk adoption failed
    disk_identity = DiskIdentity { vendor: "1b96", model: "SDS6CA264PSP9X3", serial: "372576D0016710" }
    error = Other error starting disk management: Failed to open partition at /devices/pci@3d,0/pci1de,fff8@1,1/pci1b96,2751@0/blkdev@w0014EEB3006A0600,0:wd,raw: No such file or directory (os error 2)
    file = sled-agent/config-reconciler/src/reconciler_task/external_disks.rs:703

Here is the state of the disk per prtvtoc:

22P3T7NX # prtvtoc /devices/pci@3d,0/pci1de,fff8@1,1/pci1b96,2751@0/blkdev@w0014EEB3006A0600,0:c,raw
* /devices/pci@3d,0/pci1de,fff8@1,1/pci1b96,2751@0/blkdev@w0014EEB3006A0600,0:c,raw partition map
*
* Dimensions:
*        4096 bytes/sector
*         112 sectors/track
*         224 tracks/cylinder
*       25088 sectors/cylinder
*       62292 cylinders
*       62292 accessible cylinders
*  1562781696 sectors
*  1562781696 accessible sectors
*
* Flags:
*   1: unmountable
*  10: read-only
*
*                            First       Sector      Last
* Partition  Tag  Flags      Sector       Count      Sector  Mount Directory
       0      0    00            0  1562805846  1562805845
       2      5    01            0  1562805846  1562805845
       8      1    01            0       25088       25087

Some experimentation found only certain combinations of block size and label can be adopted by sled-agent:

BlockSize Labeled Result
512 No Success: Reformatted and adopted
512 Yes Failure: Not reformatted before adoption
4k No Failure: Unable to be adopted, no :wd slice
4k Yes Success: Adopted

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in sled-agent/config-reconciler/src/reconciler_task/external_disks.rs around line 703, where the missing :wd partition causes adoption to fail. Reproduce or trace the four block-size and GPT-label combinations using prtvtoc and the reported nvmeadm format command. Done means an unlabeled 4096-byte-sector disk can be adopted, while the existing successful cases remain intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.