oxidecomputer / oxidecomputer/omicron

test failed in CI: `helios / deploy`

Open
#10,996 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Test Flake
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

This test failed on a CI run on https://github.com/oxidecomputer/omicron/pull/10972:

https://github.com/oxidecomputer/omicron/pull/10972/checks?check_run_id=91830555021

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01KZ4TC0556SH24EPYHE803264/WpXe7t9DbGWCBcpGfE6oLMABIziPfUCCsj57eISPYNggxlWR/01KZ4TCJBWTKN3HYZJJKBB2ZAP#S1020

I don't see exactly what's going on here, since there's not an obvious test failure to grab the log from. I do see a number of strange errors or warnings in the sled-agent log, such as this:

2026-08-03T23:10:51.547Z	WARN	SledAgent (RSS): sled config not yet reconciled
    error = Sled config not yet reconciled: reconcilation for disk e32c43fa-e2de-4b72-862a-603385dd3dd3 failed: Other error starting disk management: Failed to ensure filesystem 'oxp_2f15a55c-48d9-4915-9a01-1f83924e9599/crypt': Failed to initialize mountpoint at /pool/ext/2f15a55c-48d9-4915-9a01-1f83924e9599/crypt: Cannot check if mountpoint is already mounted: Command [get -Hpo value mountpoint /pool/ext/2f15a55c-48d9-4915-9a01-1f83924e9599/crypt] executed and failed with status: exit status: 1  stdout:   stderr: '/pool/ext/2f15a55c-48d9-4915-9a01-1f83924e9599/crypt': not a ZFS filesystem\n, reconcilation for dataset 0e0b1874-6272-4cd5-a234-b351d488e014 failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599, reconcilation for dataset 31e25a4a-36ea-42c7-a06a-83dd36ff94a7 failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599, reconcilation for dataset 59935688-d4e0-41ec-8ddb-196a2dd94171 failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599, reconcilation for dataset 5d9c9040-da25-450b-bdfa-c8ed919e4ce3 failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599, reconcilation for dataset 8af8c89c-949c-4361-b345-227da7a90d2f failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599, reconcilation for dataset 94249be8-2184-49bf-8114-eae9843c3fc4 failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599, reconcilation for dataset d3af50da-c947-4f39-9271-b1e14d79114b failed: could not find matching zpool oxp_2f15a55c-48d9-4915-9a01-1f83924e9599
    file = sled-agent/rack-setup/src/service.rs:557
    retry_after = 8.438915271s
    sled_address = [fd00:1122:3344:101::1]:12345

There are also a number of errors like this:

2026-08-03T23:10:52.239Z	WARN	SledAgent (ConfigReconcilerTask): failed to read M.2 slot contents
    error = no disk found in this slot
    file = sled-agent/config-reconciler/src/host_phase_2.rs:212
    slot = A
2026-08-03T23:10:52.239Z	INFO	SledAgent (ConfigReconcilerTask): reading M.2 slot to determine current contents
    file = sled-agent/config-reconciler/src/host_phase_2.rs:199
    slot = B
2026-08-03T23:10:52.239Z	WARN	SledAgent (ConfigReconcilerTask): failed to read M.2 slot contents
    error = no disk found in this slot
    file = sled-agent/config-reconciler/src/host_phase_2.rs:212
    slot = B

That's from here in the sled-agent log. All of this is odd, but suggests there's an issue with either managing zpools and / or unpacking the zone image from the CI artifact. I'm really not sure what's going on. The result though, is that the NTP zone fails to start at all.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the failed helios / deploy check for PR #10972 and the linked Buildomat logs. Inspect sled-agent/rack-setup/src/service.rs:557 and sled-agent/config-reconciler/src/host_phase_2.rs:199-212 alongside the sled-agent log entries. Done means the underlying CI failure is identified and corrected so the NTP zone starts successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.