oxidecomputer / oxidecomputer/omicron
test failed in CI: `helios / deploy`
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
This test failed on a CI run on https://github.com/oxidecomputer/omicron/pull/10972:
https://github.com/oxidecomputer/omicron/pull/10972/checks?check_run_id=91830555021
Log showing the specific test failure:
I don't see exactly what's going on here, since there's not an obvious test failure to grab the log from. I do see a number of strange errors or warnings in the sled-agent log, such as this:
2026-08-03T23:10:51.547Z WARN SledAgent (RSS): sled config not yet reconciled
error = Sled config not yet reconciled: reconcilation for disk e32c43fa-e2de-4b72-862a-603385dd3dd3 failed: Other error starting disk management: Failed to ensure filesystem 'oxp_2f15a55c-48d9-4915-9a01-1f83924e9599/crypt': Failed to initialize mountpoint at /pool/ext/2f15a55c-48d9-4915-9a01-1f83924e9599/crypt: Cannot check if mountpoint is already mounted: Command [get -Hpo value mountpoint /pool/ext/2f15a55c-48d9-4915-9a01-1f83924e9599/crypt] executed and failed with status: exit status: 1 stdout: stderr: '/pool/ext/2f15a55c-48d9-4915-9a01-1f83924e9599/crypt': not a ZFS filesystem\n, reconcilation for dataset 0e0b1874-6272-4cd5-a234-b351d488e014 failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599, reconcilation for dataset 31e25a4a-36ea-42c7-a06a-83dd36ff94a7 failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599, reconcilation for dataset 59935688-d4e0-41ec-8ddb-196a2dd94171 failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599, reconcilation for dataset 5d9c9040-da25-450b-bdfa-c8ed919e4ce3 failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599, reconcilation for dataset 8af8c89c-949c-4361-b345-227da7a90d2f failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599, reconcilation for dataset 94249be8-2184-49bf-8114-eae9843c3fc4 failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599, reconcilation for dataset d3af50da-c947-4f39-9271-b1e14d79114b failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599
file = sled-agent/rack-setup/src/service.rs:557
retry_after = 8.438915271s
sled_address = [fd00:1122:3344:101::1]:12345
There are also a number of errors like this:
2026-08-03T23:10:52.239Z WARN SledAgent (ConfigReconcilerTask): failed to read M.2 slot contents
error = no disk found in this slot
file = sled-agent/config-reconciler/src/host_phase_2.rs:212
slot = A
2026-08-03T23:10:52.239Z INFO SledAgent (ConfigReconcilerTask): reading M.2 slot to determine current contents
file = sled-agent/config-reconciler/src/host_phase_2.rs:199
slot = B
2026-08-03T23:10:52.239Z WARN SledAgent (ConfigReconcilerTask): failed to read M.2 slot contents
error = no disk found in this slot
file = sled-agent/config-reconciler/src/host_phase_2.rs:212
slot = B
That's from here in the sled-agent log. All of this is odd, but suggests there's an issue with either managing zpools and / or unpacking the zone image from the CI artifact. I'm really not sure what's going on. The result though, is that the NTP zone fails to start at all.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the failed helios / deploy check for PR #10972 and the linked Buildomat logs. Inspect sled-agent/rack-setup/src/service.rs:557 and sled-agent/config-reconciler/src/host_phase_2.rs:199-212 alongside the sled-agent log entries. Done means the underlying CI failure is identified and corrected so the NTP zone starts successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ci-cd, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100