oxidecomputer / oxidecomputer/omicron

reconfigurator stuck on newly-deployed racklette due to missing measurements

Open
#10,317 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

I deployed a racklette (with bits from a branch, but I don't think that's related) using rkadm. More specifically, from castle:

/opt/rackletteadm/bin/rkadm --toml /opt/rackletteadm/configs/dublin/dublin-rkadm.toml run --commit-a-install d13755c6ba632f518e252025113e040c33234cdf --skip-checks --skip-upgrade --commit-b-upgrade d13755c6ba632f518e252025113e040c33234cdf | tee rkadm-dublin-d13755c6ba63.out

The system came up as expected. I wanted to do a self-service update. I didn't expect that to work immediately because I never did the original "recover from mupdate", but I wanted to make sure I understood the failure mode and what would happen. So I uploaded 20.0.0-0.local+gitd13755c6ba6 and set it as the target release. At this point:

dap@atrium ~ $ oxide system update status
{
  "components_by_release_version": {
    "install dataset": 59,
    "unknown": 10,
    "20.0.0-0.local+gitd13755c6ba6": 16
  },
  "suspended": false,
  "target_release": {
    "time_requested": "2026-04-23T18:28:30.904192Z",
    "version": "20.0.0-0.local+gitd13755c6ba6"
  },
  "time_last_step_planned": "2026-04-23T17:00:03.594952Z"
}

No new blueprint has been planned:

root@oxz_switch1:~# omdb reconfigurator history
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd98:923e:55b5:104::4]:32221,[fd98:923e:55b5:102::3]:32221,[fd98:923e:55b5:101::3]:32221,[fd98:923e:55b5:103::3]:32221,[fd98:923e:55b5:104::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (252.0.0)
VERSN TIME                     BLUEPRINT      
    1 2026-04-23T17:00:03.594Z 25b77868-e93d-4423-a18e-54faf55a2696  enabled: initial blueprint from rack setup

Why not?

# omdb nexus background-tasks show blueprint_planner
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd98:923e:55b5:102::4]:12232
task: "blueprint_planner"
  configured period: every 1m
  currently executing: no
  last completed activation: iter 294, triggered by a dependent task completing
    started at 2026-04-23T18:29:20.165Z (34s ago) and ran for 305ms
    plan unchanged from parent 25b77868-e93d-4423-a18e-54faf55a2696
    note: 2/5000 blueprints in database
planning report:
* zone adds waiting on blockers
* zone adds and updates are blocked:
  - sleds have deployment units with image sources not set to Artifact:
    - sled 89d6f742-4ac3-4fd0-86f3-a021484cce8c: 15 zones
    - sled 8a92cf7d-b546-4896-9b6e-eab85112407a: 16 zones
    - sled c7b8a71c-35b5-4de2-85fc-be59ca7cd1ff: 14 zones
    - sled c873e2a3-df8e-4463-b585-109ac5d1256a: 14 zones

  - sleds have measurements with image sources not set to Artifact:
    - sled 89d6f742-4ac3-4fd0-86f3-a021484cce8c set to install dataset
    - sled 8a92cf7d-b546-4896-9b6e-eab85112407a set to install dataset
    - sled c7b8a71c-35b5-4de2-85fc-be59ca7cd1ff set to install dataset
    - sled c873e2a3-df8e-4463-b585-109ac5d1256a set to install dataset

* zone updates waiting on zone add blockers
* waiting to update top-level nexus_generation: some non-Nexus zone are not yet updated
* will ensure cockroachdb setting: "22.1"
Measurement updates:
Waiting on zone add/update blockers

This makes sense. So I went and uploaded the repo that I installed with, which was 20.0.0-0.ci+gitd13755c6ba6. Then I ran:

# oxide system update recovery-finish --system-version 20.0.0-0.ci+gitd13755c6ba6

Now:

dap@atrium ~ $ oxide system update status
{
  "components_by_release_version": {
    "unknown": 2,
    "install dataset": 59,
    "20.0.0-0.ci+gitd13755c6ba6": 24
  },
  "suspended": false,
  "target_release": {
    "time_requested": "2026-04-23T18:32:55.422689Z",
    "version": "20.0.0-0.ci+gitd13755c6ba6"
  },
  "time_last_step_planned": "2026-04-23T17:00:03.594952Z"
}

There's still only one blueprint in the history:

root@oxz_switch1:~# omdb reconfigurator history
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd98:923e:55b5:104::4]:32221,[fd98:923e:55b5:102::3]:32221,[fd98:923e:55b5:101::3]:32221,[fd98:923e:55b5:103::3]:32221,[fd98:923e:55b5:104::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (252.0.0)
VERSN TIME                     BLUEPRINT
    1 2026-04-23T17:00:03.594Z 25b77868-e93d-4423-a18e-54faf55a2696  enabled: initial blueprint from rack setup

and the planning report looks similar:

root@oxz_switch1:~# omdb nexus background-tasks show blueprint_planner
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd98:923e:55b5:102::4]:12232
task: "blueprint_planner"
  configured period: every 1m
  currently executing: no
  last completed activation: iter 307, triggered by a periodic timer firing
    started at 2026-04-23T18:34:05.118Z (1s ago) and ran for 662ms
    plan unchanged from parent 25b77868-e93d-4423-a18e-54faf55a2696
    note: 2/5000 blueprints in database
planning report:
* zone adds waiting on blockers
* zone adds and updates are blocked:
  - sleds have deployment units with image sources not set to Artifact:
    - sled 89d6f742-4ac3-4fd0-86f3-a021484cce8c: 15 zones
    - sled 8a92cf7d-b546-4896-9b6e-eab85112407a: 16 zones
    - sled c7b8a71c-35b5-4de2-85fc-be59ca7cd1ff: 14 zones
    - sled c873e2a3-df8e-4463-b585-109ac5d1256a: 14 zones

  - sleds have measurements with image sources not set to Artifact:
    - sled 89d6f742-4ac3-4fd0-86f3-a021484cce8c set to install dataset
    - sled 8a92cf7d-b546-4896-9b6e-eab85112407a set to install dataset
    - sled c7b8a71c-35b5-4de2-85fc-be59ca7cd1ff set to install dataset
    - sled c873e2a3-df8e-4463-b585-109ac5d1256a set to install dataset

* zone updates waiting on zone add blockers
* waiting to update top-level nexus_generation: some non-Nexus zone are not yet updated
* will ensure cockroachdb setting: "22.1"
Measurement updates:
Waiting on zone add/update blockers

@jgallagher pointed me to #8507 and we wondered if the problem was the missing manifest file that rkadm doesn't write out but Reconfigurator expects. Except that @sunshowers wrote in that issue:

In the interest of urgency, if the zone manifest is missing, we've chosen to synthesize one based on the existing zone images on disk rather than produce an error. That is a good enough workaround for now.

And my recollection was that this flow did work at one point (rkadm + mupdate recovery + self-service update, without having to explicit mupdate it). So I went to the inventory, which shows:

root@oxz_switch1:~# omdb db inventory collections show latest 
note: database URL not specified.  Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd98:923e:55b5:104::4]:32221,[fd98:923e:55b5:102::3]:32221,[fd98:923e:55b5:101::3]:32221,[fd98:923e:55b5:103::3]:32221,[fd98:923e:55b5:104::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (252.0.0)
collection: 68335427-bd57-4a20-9b41-dabbac5c15df
collector:  147feffe-22da-495a-8432-bae94e9e4d0c (likely a Nexus instance)
started:    2026-04-23T22:04:07.386Z
done:       2026-04-23T22:04:17.906Z
errors:     2
  error 0: MGS "http://[fd98:923e:55b5:102::2]:12225": SP SpIdentifier { slot: 14, type_: Sled }: caboose Stage0Next: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "1886596a-5b2d-4c19-8e64-75d447e2e277", "content-length": "242", "date": "Thu, 23 Apr 2026 22:04:09 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Sled, slot: 14 }: Error response from SP: failed to read data from the caboose", request_id: "1886596a-5b2d-4c19-8e64-75d447e2e277" }
  error 1: MGS "http://[fd98:923e:55b5:104::2]:12225": SP SpIdentifier { slot: 14, type_: Sled }: caboose Stage0Next: Error Response: status: 503 Service Unavailable; headers: {"content-type": "application/json", "x-request-id": "e65c1750-2379-49d3-944f-98c3156b8e3a", "content-length": "242", "date": "Thu, 23 Apr 2026 22:04:17 GMT"}; value: Error { error_code: Some("SpCommunicationFailed"), message: "error communicating with SP SpIdentifier { typ: Sled, slot: 14 }: Error response from SP: failed to read data from the caboose", request_id: "e65c1750-2379-49d3-944f-98c3156b8e3a" }

...
SLED AGENTS

sled 89d6f742-4ac3-4fd0-86f3-a021484cce8c (role = Gimlet, serial BRM23230010)
    found at:    2026-04-23T22:04:17.625Z from http://[fd98:923e:55b5:103::1]:12345
    address:     [fd98:923e:55b5:103::1]:12345
    usable hw threads:   128
    CPU family:          amd_milan
    usable memory (GiB): 895
    reservoir (GiB):     712
    physical disks:
...
    zpools
...
    datasets:
...

LEDGERED SLED CONFIG
    generation: 5
    remove_mupdate_override: None
    desired host phase 2 slot a: keep current contents
    desired host phase 2 slot b: keep current contents
    DISKS: 10
...
    DATASETS: 67
...
    ZONES: 15
        ID                                       KIND                IMAGE_SOURCE    
        00637a63-cdac-459f-a873-a164d1df4b32     internal_dns        install-dataset 
        08e79f9a-5556-45e8-beb0-4926c837487f     crucible            install-dataset 
        147feffe-22da-495a-8432-bae94e9e4d0c     nexus               install-dataset 
        149b42b8-763b-445d-a316-05b6bbb6edc4     internal_ntp        install-dataset 
        4359efec-80d1-4c91-ab65-415ae36f2764     crucible            install-dataset 
        61012e35-ef4e-4b06-89bc-6e62b194016f     crucible            install-dataset 
        69d53ce4-6b2d-42bd-aa3c-23bfa7e942d1     crucible            install-dataset 
        8626a492-365a-4bc7-ac9b-11fbb86f6c89     crucible            install-dataset 
        9ce02ce8-b78f-4693-9f09-6135cbfef31a     crucible            install-dataset 
        a3874f92-271d-46d8-b06d-e6e682d648bc     cockroach_db        install-dataset 
        a8f4aee1-28d5-41e0-89f1-6dcfab14c572     crucible            install-dataset 
        bfe05fb6-6a44-4cd9-9c10-467b3cba2bfe     crucible_pantry     install-dataset 
        c3c2eb8e-e468-4891-a985-e95ddde7e527     crucible            install-dataset 
        f1415b3e-f2fc-463c-8248-b690ce564e26     crucible            install-dataset 
        f2c75e4c-5fd7-4fde-9db0-9affe26b23c7     crucible            install-dataset 
    measurement empty
    zone image resolver status:
        zone manifest:
            path on boot disk: /pool/int/928f7fe8-d89a-4edd-885f-242f44d47999/install/zones.json
            boot disk inventory:
                manifest generated by sled-agent
                artifacts in install dataset:
                  - clickhouse.tar.gz (expected 320002184 bytes with hash b5f49d616510eaef1caf70c4d7403c278abcde1782d2b307c82c242e5f504f91): ok
                  - clickhouse_keeper.tar.gz (expected 305135542 bytes with hash be7d8456890b45e67c9e6901dab396d8f2d02fb701c9a9abcc5290d40b1275f8): ok
                  - clickhouse_server.tar.gz (expected 320001830 bytes with hash 5dcf3e9072e42d648a8e05f4bf184dddc9cdd197a4296795d55e5a845cf186de): ok
                  - cockroachdb.tar.gz (expected 164558691 bytes with hash 88e40a04e7e9008a9c75b4a1b9431da085d72bb65653037b5369d97b7133e6bd): ok
                  - crucible.tar.gz (expected 59067146 bytes with hash 5e448f9e9b2a1481d0538e284b1314a43466387f226f3b4c15d272628c361ee2): ok
                  - crucible_pantry.tar.gz (expected 44589638 bytes with hash 7fdaffcacf082357a8f1d6c2fbca28be0353f1ce9f93f41fa997e18cb77a6b55): ok
                  - external_dns.tar.gz (expected 50087695 bytes with hash e9411c659be329e365797424c5f44df8628ab7b2d857fae6ed7d2a21dc3bbc04): ok
                  - internal_dns.tar.gz (expected 50087294 bytes with hash 29f1077754db4e0bc138b28adbeaf2f2653638d8a5305900e53fa7c2bf571366): ok
                  - nexus.tar.gz (expected 225953874 bytes with hash 6a5382c7eef21074b0d029238b83eb34f8a01f318bfe2b67c84e6a600babcd6e): ok
                  - ntp.tar.gz (expected 32147381 bytes with hash cf424df626b01adefeea20b24b705b18a8d650740e0ff4d4ddad461dc05af102): ok
                  - oximeter.tar.gz (expected 64968217 bytes with hash 16c14710210512956424973d1dafc85aa8ef413f1dbe6ec1772ccd35350c7a6d): ok
                  - probe.tar.gz (expected 3109592 bytes with hash a22dca6863d0f99304e2af593bced63ccce267fc102ed2b5907031cddd22bfbd): ok
            non-boot disk status:
              - /pool/int/655c88c8-a114-4251-b598-11638274fb9c/install/zones.json (valid): valid zone manifest: 12 artifacts in manifest generated by sled-agent: 12 valid, 0 mismatched, 0 errors:
                  - clickhouse.tar.gz: valid (320002184 bytes, b5f49d616510eaef1caf70c4d7403c278abcde1782d2b307c82c242e5f504f91)
                  - clickhouse_keeper.tar.gz: valid (305135542 bytes, be7d8456890b45e67c9e6901dab396d8f2d02fb701c9a9abcc5290d40b1275f8)
                  - clickhouse_server.tar.gz: valid (320001830 bytes, 5dcf3e9072e42d648a8e05f4bf184dddc9cdd197a4296795d55e5a845cf186de)
                  - cockroachdb.tar.gz: valid (164558691 bytes, 88e40a04e7e9008a9c75b4a1b9431da085d72bb65653037b5369d97b7133e6bd)
                  - crucible.tar.gz: valid (59067146 bytes, 5e448f9e9b2a1481d0538e284b1314a43466387f226f3b4c15d272628c361ee2)
                  - crucible_pantry.tar.gz: valid (44589638 bytes, 7fdaffcacf082357a8f1d6c2fbca28be0353f1ce9f93f41fa997e18cb77a6b55)
                  - external_dns.tar.gz: valid (50087695 bytes, e9411c659be329e365797424c5f44df8628ab7b2d857fae6ed7d2a21dc3bbc04)
                  - internal_dns.tar.gz: valid (50087294 bytes, 29f1077754db4e0bc138b28adbeaf2f2653638d8a5305900e53fa7c2bf571366)
                  - nexus.tar.gz: valid (225953874 bytes, 6a5382c7eef21074b0d029238b83eb34f8a01f318bfe2b67c84e6a600babcd6e)
                  - ntp.tar.gz: valid (32147381 bytes, cf424df626b01adefeea20b24b705b18a8d650740e0ff4d4ddad461dc05af102)
                  - oximeter.tar.gz: valid (64968217 bytes, 16c14710210512956424973d1dafc85aa8ef413f1dbe6ec1772ccd35350c7a6d)
                  - probe.tar.gz: valid (3109592 bytes, a22dca6863d0f99304e2af593bced63ccce267fc102ed2b5907031cddd22bfbd)

        measurement manifest:
            path on boot disk: /pool/int/928f7fe8-d89a-4edd-885f-242f44d47999/install/measurements/measurements.json
            boot disk inventory:
                manifest generated by sled-agent
                no artifacts in install dataset (this should only be seen in simulated systems)
            non-boot disk status:
              - /pool/int/655c88c8-a114-4251-b598-11638274fb9c/install/measurements/measurements.json (valid): valid zone manifest: 0 artifacts in manifest generated by sled-agent: 0 valid, 0 mismatched, 0 errors:

        mupdate override:
            path on boot disk: /pool/int/928f7fe8-d89a-4edd-885f-242f44d47999/install/mupdate-override.json
            no override on boot disk
            non-boot disk status:
              - /pool/int/655c88c8-a114-4251-b598-11638274fb9c/install/mupdate-override.json (valid): matches boot disk (absent)
    boot disk slot: A
    slot A details:
        artifact: 153ee53b7c14ecf789be0c3cf7ca0ec98d8a175e59b866c70320015e5b2a182f (1101008896 bytes)
        image name: ci d13755c/6f14cdd 2026-04-22 04:26
        phase 2 hash: df743b3de70412e72c9bbf1d2dad38eb9df79bd386e16f9c5fd26bd8b23c9a1c
    slot B details:
        artifact: 2685505251ea95c892f3ca054834be016e9b54d137b499e1b2fd46195b36a3a3 (1101008896 bytes)
        image name: ci 550f05c/d522c04 2026-02-19 18:24
        phase 2 hash: 5cde73e6355846a3abeba958fb386b834b5500c2ffed866b8b08d16d272323f9
    last reconciled config: matches ledgered config
        no mupdate override to clear
        no orphaned datasets
        all disks reconciled successfully
        all datasets reconciled successfully
        all zones reconciled successfully
    reconciler task status: idle (finished at 2026-04-23T16:57:57.479Z after running for 15.47122336s)
    reference measurements:
        (measurement set is empty)
...

Of note:

  • "zone manifest" says "manifest generated by sled-agent" and includes a list of artifacts in the install dataset
  • "measurement manifest" says "manifest generated by sled-agent" and "no artifacts in install dataset (this should only be seen in simulated systems)"

I'm guessing what happened here is that:

  • Fully resolving the MUPdate (i.e., assigning artifacts to install-dataset images and measurements) requires valid manifest for both images and measurements.
  • When we initially did all this MUPdate resolution work, we had sled agent make up an image manifest if one was missing. At that time, there was no measurement manifest. My guess is at this time, this workflow did work. rkadm still hasn't been updated to create the manifest or mupdate override files, but that alone isn't a problem because sled agent makes up its own manifest.
  • When we landed measurements, rkadm was not updated to either deliver measurements or their manifest. Sled agent was updated to make up a manifest, but it's not a useful one in this case.
  • Now, this flow doesn't work.

and the fix is presumably the direction #8507 was going, which is to have rkadm deliver measurements and manifests.

(I believe @leftwo hit this problem the other day as well.)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the blueprint_planner report and the recovery-finish flow, then compare the inventory's zone image resolver and generated zone manifest with the sled measurements that remain set to install-dataset. Trace why the recovery does not make those sources usable for planning. Done means the planner can produce a new blueprint after this deployment and recovery sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
backend, databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.