oxidecomputer / oxidecomputer/omicron
some DNS names are too large to be looked up with our client
Open
Nobody has claimed this yet.
multi-rack
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
We tried our first self-service update on rack2 today.
It got stuck very early on:
root@oxz_switch0:~# omdb reconfigurator history --limit 2
note: database URL not specified. Will search DNS.
note: (override with --db-url or OMDB_DB_URL)
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using database URL postgresql://root@[fd00:1122:3344:109::3]:32221,[fd00:1122:3344:105::3]:32221,[fd00:1122:3344:10b::3]:32221,[fd00:1122:3344:107::3]:32221,[fd00:1122:3344:108::3]:32221/omicron?sslmode=disable
note: database schema version matches expected (196.0.0)
VERSN TIME BLUEPRINT
... (earlier history omitted)
31406 2025-10-08T20:00:06.056Z 4f033c3b-2c23-4152-9144-1d58e71458c4 enabled: sled 0c7011f7-a4bf-4daf-90cc-1c2410103300: performed 12 noop zone image source updates, sled 0c7011f7-a4bf-4daf-90cc-1c2410103300: noop updated host phase 2 to Artifact: both slot A and slot B, sled 2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa: performed 13 noop zone image source updates, sled 2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa: noop updated host phase 2 to Artifact: both slot A and slot B, sled 5f6720b8-8a31-45f8-8c94-8e699218f28b: performed 13 noop zone image source updates, sled 5f6720b8-8a31-45f8-8c94-8e699218f28b: noop updated host phase 2 to Artifact: both slot A and slot B, sled 71def415-55ad-46b4-ba88-3ca55d7fb287: performed 13 noop zone image source updates, sled 71def415-55ad-46b4-ba88-3ca55d7fb287: noop updated host phase 2 to Artifact: both slot A and slot B, sled 7b473a3b-4ec2-4b58-8376-9b3cb68d1392: performed 11 noop zone image source updates, sled 7b473a3b-4ec2-4b58-8376-9b3cb68d1392: noop updated host phase 2 to Artifact: both slot A and slot B, sled 7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c: performed 13 noop zone image source updates, sled 7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c: noop updated host phase 2 to Artifact: both slot A and slot B, sled 87c2c4fc-b0c7-4fef-a305-78f0ed265bbc: performed 13 noop zone image source updates, sled 87c2c4fc-b0c7-4fef-a305-78f0ed265bbc: noop updated host phase 2 to Artifact: both slot A and slot B, sled a2adea92-b56e-44fc-8a0d-7d63b5fd3b93: performed 13 noop zone image source updates, sled a2adea92-b56e-44fc-8a0d-7d63b5fd3b93: noop updated host phase 2 to Artifact: both slot A and slot B, sled b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88: performed 14 noop zone image source updates, sled b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88: noop updated host phase 2 to Artifact: both slot A and slot B, sled db183874-65b5-4263-a1c1-ddb2737ae0e9: performed 14 noop zone image source updates, sled db183874-65b5-4263-a1c1-ddb2737ae0e9: noop updated host phase 2 to Artifact: both slot A and slot B, sled dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd: performed 14 noop zone image source updates, sled dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd: noop updated host phase 2 to Artifact: both slot A and slot B, sled f15774c1-b8e5-434f-a493-ec43f96cba06: performed 14 noop zone image source updates, sled f15774c1-b8e5-434f-a493-ec43f96cba06: noop updated host phase 2 to Artifact: both slot A and slot B
31407 2025-10-08T20:38:30.712Z a23cb824-a19c-4d6c-b7dc-878e0eac890b enabled: update Sled 7 (BRM27230045) host phase 1 to 17.0.0-0.ci+gitb8efb9a08b3
Blueprint 31406 is the noop conversion after the MUPdate we did to get to the commit we were updating from. Blueprint 31407 is the first blueprint of the update. It configures a host OS update for sled 7:
root@oxz_switch0:~# omdb nexus blueprints diff a23cb824-a19c-4d6c-b7dc-878e0eac890b
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:104::3]:12232
from: blueprint 4f033c3b-2c23-4152-9144-1d58e71458c4
to: blueprint a23cb824-a19c-4d6c-b7dc-878e0eac890b
MODIFIED SLEDS:
sled 7b473a3b-4ec2-4b58-8376-9b3cb68d1392 (active, config generation 25 -> 26):
host phase 2 contents:
-----------------------------------------------------
slot boot image source
-----------------------------------------------------
A artifact: version 17.0.0-0.ci+git178c3c1ae84
* B - artifact: version 17.0.0-0.ci+git178c3c1ae84
└─ + artifact: version 17.0.0-0.ci+gitb8efb9a08b3
physical disks:
-------------------------------------------------
vendor model serial disposition
-------------------------------------------------
1b96 WUS4C6432DSP3X3 A079DE39 in service
1b96 WUS4C6432DSP3X3 A079DF25 in service
1b96 WUS4C6432DSP3X3 A079DF53 in service
1b96 WUS4C6432DSP3X3 A079DFFA in service
1b96 WUS4C6432DSP3X3 A079E02C in service
1b96 WUS4C6432DSP3X3 A079E02D in service
1b96 WUS4C6432DSP3X3 A079E058 in service
1b96 WUS4C6432DSP3X3 A079E20D in service
1b96 WUS4C6432DSP3X3 A079E332 in service
1b96 WUS4C6432DSP3X3 A079E4C5 in service
datasets:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
dataset name dataset id disposition quota reservation compression
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
oxp_1da520e0-2b75-428c-8f20-7b534de01ef6/crucible c2227d7c-7de4-46c5-b115-98a2cd61ea8c in service none none off
oxp_2200bb1a-40c4-4436-a4fb-d7254a72bede/crucible 5ec8bf72-c69c-4d8a-b8eb-c14b15be2953 in service none none off
oxp_2bbb6a2e-35bc-4c57-ae37-fb59365c2e1d/crucible 84f3e06c-8b98-4250-be94-0672319f27f9 in service none none off
oxp_363719bd-5ed1-4546-a283-30f916e0b956/crucible 3be05156-2dbc-4807-9c72-9872bf687bef in service none none off
oxp_366f1271-7668-4aaa-b767-4f71d76cb940/crucible 8d3118a5-7d5e-4c1a-84e8-e6ea7fbf563e in service none none off
oxp_786edb5b-7542-4c22-a9cc-4c78f5ad5397/crucible c3ae692a-4bea-416e-875c-dd9caa8422be in service none none off
oxp_8a0016f3-3cdc-4b4c-b1d3-5334dd365465/crucible 0b8197fd-18ad-4cf8-a48c-c1b5e47828c7 in service none none off
oxp_8a955c00-1918-4bf0-bc15-7e766758cefe/crucible 7cdb3c11-f792-4195-9152-16a948fb271e in service none none off
oxp_a55e38ca-0e11-4793-989a-5741611726ea/crucible 0cc37729-1489-4c8a-b36f-0c145563d803 in service none none off
oxp_eb3bfc78-6be5-484d-865f-e01c5cbcab46/crucible 7fa766ed-942a-478e-b61d-b8f28584eff7 in service none none off
oxp_1da520e0-2b75-428c-8f20-7b534de01ef6/crypt/zone cc7b3ca3-9524-4a82-96b3-609b69e86e0f in service none none off
oxp_2200bb1a-40c4-4436-a4fb-d7254a72bede/crypt/zone 9b648ce3-65a3-4d19-9933-fea673629df4 in service none none off
oxp_2bbb6a2e-35bc-4c57-ae37-fb59365c2e1d/crypt/zone 45ec7703-7de1-4b64-8c46-6b4405410e28 in service none none off
oxp_363719bd-5ed1-4546-a283-30f916e0b956/crypt/zone 0235457d-1353-4e88-91e3-2136dffb082f in service none none off
oxp_366f1271-7668-4aaa-b767-4f71d76cb940/crypt/zone 0ccc1618-687c-4b2f-a33b-4ee3a5cb39dc in service none none off
oxp_786edb5b-7542-4c22-a9cc-4c78f5ad5397/crypt/zone 276fe6de-0002-487c-bb77-f26e1be8b6d3 in service none none off
oxp_8a0016f3-3cdc-4b4c-b1d3-5334dd365465/crypt/zone 19d9ddf8-0ba7-484e-94aa-ec3861ca6db2 in service none none off
oxp_8a955c00-1918-4bf0-bc15-7e766758cefe/crypt/zone f5bdc2e0-4080-41e7-8fd6-5093cddb7abd in service none none off
oxp_a55e38ca-0e11-4793-989a-5741611726ea/crypt/zone 624aa518-1cfa-4936-9125-77200836e8c2 in service none none off
oxp_eb3bfc78-6be5-484d-865f-e01c5cbcab46/crypt/zone 56589a9b-8e1e-4944-8094-145d22f8505e in service none none off
oxp_786edb5b-7542-4c22-a9cc-4c78f5ad5397/crypt/zone/oxz_crucible_16f1837a-f41f-4394-bfc5-b04359d777ce e026f065-cbab-42a4-ac84-d2ca1c2468b8 in service none none off
oxp_a55e38ca-0e11-4793-989a-5741611726ea/crypt/zone/oxz_crucible_275e54e9-62b6-4262-9798-be2768effa1c 8dd67f2f-4074-493d-808e-1f2a5ebec6fe in service none none off
oxp_2200bb1a-40c4-4436-a4fb-d7254a72bede/crypt/zone/oxz_crucible_337c4cf5-8c10-4284-b541-a12b22551d89 d57bb0dd-cdf4-4ad2-8ff9-d71ee06088df in service none none off
oxp_366f1271-7668-4aaa-b767-4f71d76cb940/crypt/zone/oxz_crucible_4778fc0d-6a75-4e94-b95b-08f6e1977c0e d06501c3-608b-4af1-a89f-9e87f05f6290 in service none none off
oxp_8a955c00-1918-4bf0-bc15-7e766758cefe/crypt/zone/oxz_crucible_65eb924b-b72c-4e4f-ae39-1f431016d2a8 5ec1a47a-abdb-458e-b21e-6a91be1c9012 in service none none off
oxp_1da520e0-2b75-428c-8f20-7b534de01ef6/crypt/zone/oxz_crucible_b5e9c0c0-d63c-4f31-bd14-45f737d57e28 96f0ca6f-6d58-4ae9-bb52-a24b7eac86ab in service none none off
oxp_eb3bfc78-6be5-484d-865f-e01c5cbcab46/crypt/zone/oxz_crucible_cfad3cc7-4f74-479f-8c0a-f33fe2b8216a cb27abd2-54e3-4ada-94af-b3cd8b7e3a50 in service none none off
oxp_363719bd-5ed1-4546-a283-30f916e0b956/crypt/zone/oxz_crucible_de0ed7c4-4e7b-4d4a-8e35-744b1cb07624 f4e40316-430a-4663-88ba-31f7b57113c4 in service none none off
oxp_2bbb6a2e-35bc-4c57-ae37-fb59365c2e1d/crypt/zone/oxz_crucible_e6c135c7-d47c-4ba4-9b0f-749d3f00ac9e 549132a1-5adb-4922-98a2-51774aa005f7 in service none none off
oxp_8a0016f3-3cdc-4b4c-b1d3-5334dd365465/crypt/zone/oxz_crucible_f50d5a7f-458e-420b-af48-7264cf3ea605 71df7bef-4103-48c3-894a-53acde5228d0 in service none none off
oxp_1da520e0-2b75-428c-8f20-7b534de01ef6/crypt/zone/oxz_ntp_954bb994-8892-4ca8-9006-7901ff799e27 71537ff5-c25f-4783-b0d7-41af0df9fe39 in service none none off
oxp_1da520e0-2b75-428c-8f20-7b534de01ef6/crypt/debug 55df4e0c-3a2b-4e9b-b596-ce60ed186a67 in service 100 GiB none gzip-9
oxp_2200bb1a-40c4-4436-a4fb-d7254a72bede/crypt/debug 25846e1a-e811-432a-96e5-9266a1daa6b7 in service 100 GiB none gzip-9
oxp_2bbb6a2e-35bc-4c57-ae37-fb59365c2e1d/crypt/debug 3cad72ad-b5eb-4375-85cf-55daabdf1093 in service 100 GiB none gzip-9
oxp_363719bd-5ed1-4546-a283-30f916e0b956/crypt/debug 98f5287a-0cb4-4418-8a7b-ca4e051d9940 in service 100 GiB none gzip-9
oxp_366f1271-7668-4aaa-b767-4f71d76cb940/crypt/debug 7efda33a-ff0a-42f8-99f6-f5acd361a80e in service 100 GiB none gzip-9
oxp_786edb5b-7542-4c22-a9cc-4c78f5ad5397/crypt/debug bc0168b4-46b6-4cd7-aa65-3c0879d4aa1b in service 100 GiB none gzip-9
oxp_8a0016f3-3cdc-4b4c-b1d3-5334dd365465/crypt/debug a75f1671-593c-4d26-a296-6158221f0222 in service 100 GiB none gzip-9
oxp_8a955c00-1918-4bf0-bc15-7e766758cefe/crypt/debug 493c41a5-dfe8-4e07-b1c6-b5abb6293064 in service 100 GiB none gzip-9
oxp_a55e38ca-0e11-4793-989a-5741611726ea/crypt/debug af08c481-b6de-41ba-ab05-75fd5bdb4e1f in service 100 GiB none gzip-9
oxp_eb3bfc78-6be5-484d-865f-e01c5cbcab46/crypt/debug 7f5d5e90-aaa5-4b59-b04d-58046a73fe2c in service 100 GiB none gzip-9
omicron zones:
-----------------------------------------------------------------------------------------------------------------------------------------
zone type zone id image source disposition underlay IP
-----------------------------------------------------------------------------------------------------------------------------------------
crucible 16f1837a-f41f-4394-bfc5-b04359d777ce artifact: version 17.0.0-0.ci+git178c3c1ae84 in service fd00:1122:3344:127::27
crucible 275e54e9-62b6-4262-9798-be2768effa1c artifact: version 17.0.0-0.ci+git178c3c1ae84 in service fd00:1122:3344:127::2a
crucible 337c4cf5-8c10-4284-b541-a12b22551d89 artifact: version 17.0.0-0.ci+git178c3c1ae84 in service fd00:1122:3344:127::23
crucible 4778fc0d-6a75-4e94-b95b-08f6e1977c0e artifact: version 17.0.0-0.ci+git178c3c1ae84 in service fd00:1122:3344:127::26
crucible 65eb924b-b72c-4e4f-ae39-1f431016d2a8 artifact: version 17.0.0-0.ci+git178c3c1ae84 in service fd00:1122:3344:127::29
crucible b5e9c0c0-d63c-4f31-bd14-45f737d57e28 artifact: version 17.0.0-0.ci+git178c3c1ae84 in service fd00:1122:3344:127::22
crucible cfad3cc7-4f74-479f-8c0a-f33fe2b8216a artifact: version 17.0.0-0.ci+git178c3c1ae84 in service fd00:1122:3344:127::2b
crucible de0ed7c4-4e7b-4d4a-8e35-744b1cb07624 artifact: version 17.0.0-0.ci+git178c3c1ae84 in service fd00:1122:3344:127::25
crucible e6c135c7-d47c-4ba4-9b0f-749d3f00ac9e artifact: version 17.0.0-0.ci+git178c3c1ae84 in service fd00:1122:3344:127::24
crucible f50d5a7f-458e-420b-af48-7264cf3ea605 artifact: version 17.0.0-0.ci+git178c3c1ae84 in service fd00:1122:3344:127::28
internal_ntp 954bb994-8892-4ca8-9006-7901ff799e27 artifact: version 17.0.0-0.ci+git178c3c1ae84 in service fd00:1122:3344:127::21
COCKROACHDB SETTINGS:
state fingerprint::::::::::::::::: d4d87aa2ad877a4cc2fddd0573952362739110de (unchanged)
cluster.preserve_downgrade_option: "22.1" (unchanged)
METADATA:
internal DNS version::: 45 (unchanged)
external DNS version::: 150 (unchanged)
target release min gen: 2 (unchanged)
nexus gen:::::::::::::: 1 (unchanged)
OXIMETER SETTINGS:
generation: 5 (unchanged)
read from:: SingleNode (unchanged)
CLICKHOUSE CLUSTER CONFIG:
generation::::::::::::::::::::::::::::::::::::: 2 (unchanged)
max used server id::::::::::::::::::::::::::::: 3 (unchanged)
max used keeper id::::::::::::::::::::::::::::: 5 (unchanged)
cluster name::::::::::::::::::::::::::::::::::: oximeter_cluster (unchanged)
cluster secret::::::::::::::::::::::::::::::::: 7443e06e-df28-4942-8188-a1baf7eeaafe (unchanged)
highest seen keeper leader committed log index: 3226523392 (unchanged)
clickhouse keepers at generation 2:
------------------------------------------------
zone id keeper id
------------------------------------------------
36e65da6-3ebe-43f3-8239-75c969b5e558 1
73404218-b57f-48d7-87c7-824fea56a988 2
95363b23-6e50-4bf1-9fbe-f16bdfc021c2 3
b251c2b6-e1c4-4874-8a7d-236eda8bb211 4
c7c01469-825e-494a-ad4a-3423f353ccc4 5
clickhouse servers at generation 2:
------------------------------------------------
zone id server id
------------------------------------------------
69cce3c6-e957-4e12-b3fb-34f5aa6ecc9f 1
a1336084-d8df-4730-a805-4cf420a5bc06 2
f9065d53-df45-482a-9c92-dd13098e5d82 3
PENDING MGS UPDATES:
Pending MGS-managed updates (all baseboards):
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
sp_type slot part_number serial_number artifact_hash artifact_version details
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ sled 7 913-0000019 BRM27230045 3a6940e9917d578eb8c9c81e491f37885fbc044e37741fa2fe36a278e0012747 17.0.0-0.ci+gitb8efb9a08b3 HostPhase1(PendingMgsUpdateHostPhase1Details { expected_active_phase_1_slot: A, expected_boot_disk: A, expected_active_phase_1_hash: ArtifactHash("fc543f65851ad081f4f2c41148e57cdf498435df873c86807c1a478bb308ef4f"), expected_active_phase_2_hash: ArtifactHash("71648440d5f7362919a077f681203410129dff1bec35d8aaecff61372e0da7cd"), expected_inactive_phase_1_hash: ArtifactHash("fc543f65851ad081f4f2c41148e57cdf498435df873c86807c1a478bb308ef4f"), expected_inactive_phase_2_hash: ArtifactHash("ff9e7fd4c0b524be22a7db1081ff406be995459362ff18845c38c04782a66302"), sled_agent_address: [fd00:1122:3344:127::1]:12345 })
The first thing this should do is update phase 2, and indeed host OS phase 2 has been updated on that sled:
root@oxz_switch0:~# omdb db inventory collections show latest
...
sled 7b473a3b-4ec2-4b58-8376-9b3cb68d1392 (role = Gimlet, serial BRM27230045)
found at: 2025-10-08T23:19:15.371Z from http://[fd00:1122:3344:127::1]:12345
address: [fd00:1122:3344:127::1]:12345
usable hw threads: 128
CPU family: amd_milan
usable memory (GiB): 1011
reservoir (GiB): 809
...
LEDGERED SLED CONFIG
generation: 26
remove_mupdate_override: None
desired host phase 2 slot a: artifact 71648440d5f7362919a077f681203410129dff1bec35d8aaecff61372e0da7cd
desired host phase 2 slot b: artifact ff9e7fd4c0b524be22a7db1081ff406be995459362ff18845c38c04782a66302
...
boot disk slot: A
slot A details:
artifact: 71648440d5f7362919a077f681203410129dff1bec35d8aaecff61372e0da7cd (1048580096 bytes)
image name: ci 178c3c1/b239531 2025-10-07 20:53
phase 2 hash: e603355a41433a3d6849b9c976fe2451248e0708b3518b59b0df5e356a98ea7c
slot B details:
artifact: ff9e7fd4c0b524be22a7db1081ff406be995459362ff18845c38c04782a66302 (1048580096 bytes)
image name: ci b8efb9a/b239531 2025-10-07 21:31
phase 2 hash: 8dbc16c42e982209487d8ffdcb735c110dae6f6a490e1a9b94ec26ef12d7b611
last reconciled config: matches ledgered config
no mupdate override to clear
no orphaned datasets
all disks reconciled successfully
all datasets reconciled successfully
all zones reconciled successfully
reconciler task status: idle (finished at 2025-10-08T20:38:48.616Z after running for 17.320001131s)
The key bit there is that boot disk is slot A and slot B has image b8efb9a, which is the commit we were trying to update to. So we successfully propagated phase 2 to this sled.
So what's going on with the update to phase 1? We can ask Nexus how that's going:
root@oxz_switch0:~# omdb nexus background-tasks show blueprint_executor
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:104::3]:12232
task: "blueprint_executor"
configured period: every 1m
currently executing: no
last completed activation: iter 269, triggered by a periodic timer firing
started at 2025-10-08T23:22:11.000Z (29s ago) and ran for 8583ms
target blueprint: a23cb824-a19c-4d6c-b7dc-878e0eac890b
execution: enabled
status: completed (15 steps)
error: (none)
root@oxz_switch0:~# omdb nexus background-tasks print-report blueprint_executor
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:104::3]:12232
[Oct 08 23:22:11] Running ( 1/15) Ensure external networking resources
[Oct 08 23:22:11] Completed ( 1/15) Ensure external networking resources: after 93.64ms
[Oct 08 23:22:11] Running ( 2/15) Fetch sled list
[Oct 08 23:22:11] Completed ( 2/15) Fetch sled list: after 89.02ms
[Oct 08 23:22:11] Running ( 3/15) Ensure db_metadata_nexus_state records exist
[Oct 08 23:22:11] Completed ( 3/15) Ensure db_metadata_nexus_state records exist: after 51.67ms
[Oct 08 23:22:11] Running ( 4/15) Deploy sled configs
[Oct 08 23:22:16] Completed ( 4/15) Deploy sled configs: after 5.52s
[Oct 08 23:22:16] Running ( 5/15) Plumb service firewall rules
[Oct 08 23:22:17] Completed ( 5/15) Plumb service firewall rules: after 866.92ms
[Oct 08 23:22:17] Running ( 6/15) Deploy DNS records
[Oct 08 23:22:18] Completed ( 6/15) Deploy DNS records: after 422.26ms
[Oct 08 23:22:18] Running ( 7/15) Cleanup expunged zones
[Oct 08 23:22:18] Completed ( 7/15) Cleanup expunged zones: after 33.50µs
[Oct 08 23:22:18] Running ( 8/15) Decommission sleds
[Oct 08 23:22:18] Completed ( 8/15) Decommission sleds: after 327.40ms
[Oct 08 23:22:18] Running ( 9/15) Decommission expunged disks
[Oct 08 23:22:19] Completed ( 9/15) Decommission expunged disks: after 681.23ms
[Oct 08 23:22:19] Running (10/15) Deploy clickhouse cluster nodes
[Oct 08 23:22:19] Completed (10/15) Deploy clickhouse cluster nodes: after 358.94ms
[Oct 08 23:22:19] Running (11/15) Deploy single-node clickhouse cluster
[Oct 08 23:22:19] Completed (11/15) Deploy single-node clickhouse cluster: after 49.79ms
[Oct 08 23:22:19] Running (12/15) Mark support bundles as failed if they rely on an expunged disk or sled
[Oct 08 23:22:19] Completed (12/15) Mark support bundles as failed if they rely on an expunged disk or sled: after 62.24ms with message: support bundle expunge report: SupportBundleExpungementReport { bundles_failed_missing_datasets: 0, bundles_deleted_missing_datasets: 0, bundles_failing_missing_nexus: 0, bundles_reassigned: 0 }
[Oct 08 23:22:19] Running (13/15) Reassign sagas
[Oct 08 23:22:19] Completed (13/15) Reassign sagas: after 46.58ms
[Oct 08 23:22:19] Running (14/15) Ensure CockroachDB settings
[Oct 08 23:22:19] Completed (14/15) Ensure CockroachDB settings: after 11.37ms
[Oct 08 23:22:19] Running (15/15) Kick off MGS-managed updates
[Oct 08 23:22:19] Completed (15/15) Kick off MGS-managed updates: after 6.54µs
So blueprint execution is at least working. Let's ask about MGS-driven updates:
# omdb nexus mgs-updates
note: Nexus URL not specified. Will pick one from DNS.
note: using DNS server for subnet fd00:1122:3344::/48
note: (if this is not right, use --dns-server to specify an alternate DNS server)
note: using Nexus URL http://[fd00:1122:3344:104::3]:12232
recent completed attempts:
2025-10-08T23:07:49.750Z to 2025-10-08T23:07:49.827Z (took 77ms): serial BRM27230045
attempt#: 150
version: 17.0.0-0.ci+gitb8efb9a08b3
hash: 3a6940e9917d578eb8c9c81e491f37885fbc044e37741fa2fe36a278e0012747
result: Err("failed to fetch artifact: no repo depot clients available")
...
2025-10-08T23:22:51.943Z to 2025-10-08T23:22:52.013Z (took 69ms): serial BRM27230045
attempt#: 165
version: 17.0.0-0.ci+gitb8efb9a08b3
hash: 3a6940e9917d578eb8c9c81e491f37885fbc044e37741fa2fe36a278e0012747
result: Err("failed to fetch artifact: no repo depot clients available")
currently in progress:
waiting for retry:
serial BRM27230045: will try again at 2025-10-08 23:23:52.013361249 UTC (attempt 166)
That's really strange. Looking at the code: ArtifactCache creates one Repo Depot client for each backend reported by qorb. Qorb must be reporting no backends. We spent some time trying to figure out what qorb was doing (was it trying to do anything? was it sending DNS queries? getting responses? seeing any errors?) but there are no logs (see oxidecomputer/qorb#111) and the DTrace probes seemed specific to connection management, not DNS.
We were able to confirm that with dig, all of the internal DNS servers are reporting all of the repo depot backends:
# dig -t SRV _repo-depot._tcp.control-plane.oxide.internal. @fd00:1122:3344:1::1
; <<>> DiG 9.18.14 <<>> -t SRV _repo-depot._tcp.control-plane.oxide.internal. @fd00:1122:3344:1::1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52873
;; flags: qr aa rd; QUERY: 1, ANSWER: 12, AUTHORITY: 0, ADDITIONAL: 12
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;_repo-depot._tcp.control-plane.oxide.internal. IN SRV
;; ANSWER SECTION:
_repo-depot._tcp.control-plane.oxide.internal. 0 IN SRV 0 0 12348 0c7011f7-a4bf-4daf-90cc-1c2410103300.sled.control-plane.oxide.internal.
_repo-depot._tcp.control-plane.oxide.internal. 0 IN SRV 0 0 12348 2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa.sled.control-plane.oxide.internal.
_repo-depot._tcp.control-plane.oxide.internal. 0 IN SRV 0 0 12348 5f6720b8-8a31-45f8-8c94-8e699218f28b.sled.control-plane.oxide.internal.
_repo-depot._tcp.control-plane.oxide.internal. 0 IN SRV 0 0 12348 71def415-55ad-46b4-ba88-3ca55d7fb287.sled.control-plane.oxide.internal.
_repo-depot._tcp.control-plane.oxide.internal. 0 IN SRV 0 0 12348 7b473a3b-4ec2-4b58-8376-9b3cb68d1392.sled.control-plane.oxide.internal.
_repo-depot._tcp.control-plane.oxide.internal. 0 IN SRV 0 0 12348 7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c.sled.control-plane.oxide.internal.
_repo-depot._tcp.control-plane.oxide.internal. 0 IN SRV 0 0 12348 87c2c4fc-b0c7-4fef-a305-78f0ed265bbc.sled.control-plane.oxide.internal.
_repo-depot._tcp.control-plane.oxide.internal. 0 IN SRV 0 0 12348 a2adea92-b56e-44fc-8a0d-7d63b5fd3b93.sled.control-plane.oxide.internal.
_repo-depot._tcp.control-plane.oxide.internal. 0 IN SRV 0 0 12348 b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88.sled.control-plane.oxide.internal.
_repo-depot._tcp.control-plane.oxide.internal. 0 IN SRV 0 0 12348 db183874-65b5-4263-a1c1-ddb2737ae0e9.sled.control-plane.oxide.internal.
_repo-depot._tcp.control-plane.oxide.internal. 0 IN SRV 0 0 12348 dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd.sled.control-plane.oxide.internal.
_repo-depot._tcp.control-plane.oxide.internal. 0 IN SRV 0 0 12348 f15774c1-b8e5-434f-a493-ec43f96cba06.sled.control-plane.oxide.internal.
;; ADDITIONAL SECTION:
0c7011f7-a4bf-4daf-90cc-1c2410103300.sled.control-plane.oxide.internal. 0 IN AAAA fd00:1122:3344:104::1
2707b587-9c7f-4fb0-a7af-37c3b7a9a0fa.sled.control-plane.oxide.internal. 0 IN AAAA fd00:1122:3344:103::1
5f6720b8-8a31-45f8-8c94-8e699218f28b.sled.control-plane.oxide.internal. 0 IN AAAA fd00:1122:3344:109::1
71def415-55ad-46b4-ba88-3ca55d7fb287.sled.control-plane.oxide.internal. 0 IN AAAA fd00:1122:3344:10b::1
7b473a3b-4ec2-4b58-8376-9b3cb68d1392.sled.control-plane.oxide.internal. 0 IN AAAA fd00:1122:3344:127::1
7b862eb6-7f50-4c2f-b9a6-0d12ac913d3c.sled.control-plane.oxide.internal. 0 IN AAAA fd00:1122:3344:101::1
87c2c4fc-b0c7-4fef-a305-78f0ed265bbc.sled.control-plane.oxide.internal. 0 IN AAAA fd00:1122:3344:108::1
a2adea92-b56e-44fc-8a0d-7d63b5fd3b93.sled.control-plane.oxide.internal. 0 IN AAAA fd00:1122:3344:102::1
b886b58a-1e3f-4be1-b9f2-0c2e66c6bc88.sled.control-plane.oxide.internal. 0 IN AAAA fd00:1122:3344:106::1
db183874-65b5-4263-a1c1-ddb2737ae0e9.sled.control-plane.oxide.internal. 0 IN AAAA fd00:1122:3344:107::1
dd83e75a-1edf-4aa1-89a0-cd8b2091a7cd.sled.control-plane.oxide.internal. 0 IN AAAA fd00:1122:3344:10a::1
f15774c1-b8e5-434f-a493-ec43f96cba06.sled.control-plane.oxide.internal. 0 IN AAAA fd00:1122:3344:105::1
;; Query time: 0 msec
;; SERVER: fd00:1122:3344:1::1#53(fd00:1122:3344:1::1) (UDP)
;; WHEN: Wed Oct 08 23:26:26 UTC 2025
;; MSG SIZE rcvd: 1524
We spot-checked several of these and found that they were at least running (i.e., we could hit them with curl). (This shouldn't matter -- we'd get a different error if the problem were with the servers themselves.)
We used snoop to try to see what was going on and found some surprising results.
Here's the result of snooping from a Nexus zone:
# snoop -d oxControlService12 port 53 | tee dap-snoop.out
...
oxz_nexus_65a11c18-7f59-41ac-b9e7-680627f996e7.local -> fd00:1122:3344:2::1 DNS C _repo-depot._tcp.control-plane.oxide.internal. IN SRV ?
oxz_nexus_65a11c18-7f59-41ac-b9e7-680627f996e7.local -> fd00:1122:3344:3::1 DNS C _repo-depot._tcp.control-plane.oxide.internal. IN SRV ?
oxz_nexus_65a11c18-7f59-41ac-b9e7-680627f996e7.local -> fd00:1122:3344:1::1 DNS C _repo-depot._tcp.control-plane.oxide.internal. IN SRV ?
oxz_nexus_65a11c18-7f59-41ac-b9e7-680627f996e7.local -> fd00:1122:3344:2::1 DNS C _mgs._tcp.control-plane.oxide.internal. IN SRV ?
oxz_nexus_65a11c18-7f59-41ac-b9e7-680627f996e7.local -> fd00:1122:3344:1::1 DNS C _mgs._tcp.control-plane.oxide.internal. IN SRV ?
oxz_nexus_65a11c18-7f59-41ac-b9e7-680627f996e7.local -> fd00:1122:3344:3::1 DNS C _mgs._tcp.control-plane.oxide.internal. IN SRV ?
...
We see the queries go out, but we don't see responses come back. I am not clear if they were actually not coming back, or if they were being fragmented, or if something else was going on.
Here's the results from snooping from an internal DNS zone using snoop -o dapsnoop.out -d oxControlService0 port 53 while both dig (at T=3.03s) and Nexus queried for repo depot:
We don't see responses. That's despite the fact that my dig invocation succeeded!
By contrast, for MGS, we see the responses:
I'm not sure how we made this leap, but we started considering UDP vs. TCP for DNS and eDNS and whether size limits might be a problem. That's when we noticed this from the dig output above:
;; MSG SIZE rcvd: 1524
That's just a bit bigger than some important limits (1500 bytes typical MTU, and 1232 bytes standard DNS size). It looks to me like hickory-dns uses a buffer size of 1232 bytes:
https://github.com/hickory-dns/hickory-dns/blob/527c9f470a418cf6b92da902ea0aaa5749963d59/crates/proto/src/xfer/dns_handle.rs#L23-L27
It looks like we're hitting the same problem described in #6415 that I described in more detail here. There, we ran into a problem at 512 bytes and the fix was to enable eDNS, not to use eDNS, but because it caused hickory-dns to use a larger buffer size. We seem now to be over that buffer size.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the DNS-based lookup used by omdb reconfigurator history and omdb nexus blueprints diff; the issue names no source file or test. Trace the client’s DNS-name handling and identify where the oversized name fails. Done means the client can resolve the names involved in this self-service update path without requiring the explicit override options.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100