oxidecomputer / oxidecomputer/omicron

a4x2: rack setup request timed out

Open
#10,047 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

I ran a4x2 and found at least one rack setup request failed unexpectedly:

21:23:10.726Z INFO SledAgent (RSS): Failed to handoff to nexus: Error Response: status: 404 Not Found; headers: {"content-type": "application/json", "x-request-id": "835951c8-c445-4b5b-9fa2-b414c6993afc", "content-length": "173", "date": "Thu, 12 Mar 2026 21:23:10 GMT"}; value: Error { error_code: Some("ObjectNotFound"), message: "not found: rack with id \"13f51333-f4fe-4b17-9692-08c2d4b8be59\"", request_id: "835951c8-c445-4b5b-9fa2-b414c6993afc" }
    file = sled-agent/src/rack_setup/service.rs:1061
21:24:36.525Z INFO SledAgent (RSS): Failed to handoff to nexus: Communication Error: error sending request for url (http://[fd00:17:1:d02::5]:12232/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete)
    file = sled-agent/src/rack_setup/service.rs:1061
21:25:48.770Z INFO SledAgent (RSS): Handoff to Nexus is complete
    file = sled-agent/src/rack_setup/service.rs:1071

Looking through the Nexus log for a rack initialization request, I found this one:

{"msg":"updating rack subnet for rack 13f51333-f4fe-4b17-9692-08c2d4b8be59 to Some(\n    V6(\n        Ipv6Network {\n            addr: fd00:17:1:d00::,\n            prefix: 56,\n        },\n    ),\n)","v":0,"name":"nexus","level":20,"time":"2026-03-12T21:23:40.663575294Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf"}
{"msg":"Using automatic external switchport discovery","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:23:40.6729458Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/src/app/rack.rs:309"}
{"msg":"discovered ports for switch1: [\n    Internal(\n        Internal(\n            \"int0\",\n        ),\n    ),\n    Rear(\n        Rear(\n            \"rear0\",\n        ),\n    ),\n    Rear(\n        Rear(\n            \"rear1\",\n        ),\n    ),\n    Rear(\n        Rear(\n            \"rear2\",\n        ),\n    ),\n    Rear(\n        Rear(\n            \"rear3\",\n        ),\n    ),\n    Qsfp(\n        Qsfp(\n            \"qsfp0\",\n        ),\n    ),\n    Qsfp(\n        Qsfp(\n            \"qsfp1\",\n        ),\n    ),\n]","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:23:40.689104647Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/src/app/rack.rs:329"}
{"msg":"populating ports for switch1: [\n    Name(\n        \"qsfp0\",\n    ),\n    Name(\n        \"qsfp1\",\n    ),\n]","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:23:40.689211416Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/src/app/rack.rs:339"}
{"msg":"discovered ports for switch0: [\n    Internal(\n        Internal(\n            \"int0\",\n        ),\n    ),\n    Rear(\n        Rear(\n            \"rear0\",\n        ),\n    ),\n    Rear(\n        Rear(\n            \"rear1\",\n        ),\n    ),\n    Rear(\n        Rear(\n            \"rear2\",\n        ),\n    ),\n    Rear(\n        Rear(\n            \"rear3\",\n        ),\n    ),\n    Qsfp(\n        Qsfp(\n            \"qsfp0\",\n        ),\n    ),\n    Qsfp(\n        Qsfp(\n            \"qsfp1\",\n        ),\n    ),\n]","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:23:47.723656048Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/src/app/rack.rs:329"}
{"msg":"populating ports for switch0: [\n    Name(\n        \"qsfp0\",\n    ),\n    Name(\n        \"qsfp1\",\n    ),\n]","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:23:47.723990956Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/src/app/rack.rs:339"}
{"msg":"Recording Rack Network Configuration","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:23:56.054366246Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/src/app/rack.rs:365"}
{"msg":"no matching bgp config found","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:23:58.943355431Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/bgp.rs:134"}
{"msg":"no matching bgp config found","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:23:59.238610897Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/bgp.rs:134"}
{"msg":"inserted blueprint","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:27.493101917Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/deployment.rs:602","blueprint_id":"386b1df2-665d-40e7-8e49-70bdbd8be9fd"}
{"msg":"Inserted networking records for nexus service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.206179825Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:696"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.206590379Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.206769173Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for oximeter service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.206792365Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.206804826Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible_pantry service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.206816657Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for cockroach_db service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.20684633Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for internal_dns service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.206858731Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.206869632Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"Inserted networking records for boundary_ntp service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.244739406Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:696"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.244859666Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for cockroach_db service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.244876087Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.244888578Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.244902789Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"Inserted networking records for nexus service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.283658287Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:696"}
{"msg":"No networking records needed for cockroach_db service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.283725263Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.283738024Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.283749305Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for internal_ntp service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.283760796Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"Inserted networking records for external_dns service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.347839239Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:696"}
{"msg":"No networking records needed for internal_dns service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.347917165Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.347939457Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for clickhouse service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.347957368Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible_pantry service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.34797855Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.347994531Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.348010953Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible_pantry service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.348029504Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for cockroach_db service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.348048076Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for internal_dns service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.348066237Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.348083409Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.34810262Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"Inserted networking records for boundary_ntp service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.380997203Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:696"}
{"msg":"No networking records needed for cockroach_db service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.38107943Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"Inserted networking records for external_dns service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.424309308Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:696"}
{"msg":"No networking records needed for crucible service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.424380313Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:649"}
{"msg":"Inserted networking records for nexus service","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.457867065Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:696"}
{"msg":"Inserted service networking records","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.457971294Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:875"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: 0326f511-33d7-4a6b-9ff7-551c8d8fa46e (physical_disk),\n        time_created: 2026-03-12T21:23:40.640783Z,\n        time_modified: 2026-03-12T21:23:40.640783Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g0_4\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: ae8e5fb5-9622-404b-8923-cff060b0c48a (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.457995176Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: 59d4e545-6e26-45cb-b6bd-50a1abc91404 (physical_disk),\n        time_created: 2026-03-12T21:23:40.640784Z,\n        time_modified: 2026-03-12T21:23:40.640784Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g0_2\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: ae8e5fb5-9622-404b-8923-cff060b0c48a (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.507959711Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: 5a1e8610-3bb8-49f3-bbb3-9ea7b8c13b87 (physical_disk),\n        time_created: 2026-03-12T21:23:40.640784Z,\n        time_modified: 2026-03-12T21:23:40.640784Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g0_0\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: ae8e5fb5-9622-404b-8923-cff060b0c48a (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.527467626Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: 88cf0bd2-6503-4e47-b04c-f7244497981a (physical_disk),\n        time_created: 2026-03-12T21:23:40.640784Z,\n        time_modified: 2026-03-12T21:23:40.640784Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g0_3\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: ae8e5fb5-9622-404b-8923-cff060b0c48a (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.554362652Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: a2661149-da3f-42af-abbb-c5e4f8f6179a (physical_disk),\n        time_created: 2026-03-12T21:23:40.640784Z,\n        time_modified: 2026-03-12T21:23:40.640784Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g0_1\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: ae8e5fb5-9622-404b-8923-cff060b0c48a (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.577395008Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: 0b53dcaa-3b38-441e-9959-9ef8e5adb0bb (physical_disk),\n        time_created: 2026-03-12T21:23:40.640784Z,\n        time_modified: 2026-03-12T21:23:40.640784Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g1_2\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: 09d5c82d-07ee-4f9a-9985-e2609f9a2212 (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:24:28.59422253Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"request handling cancelled (client disconnected)","v":0,"name":"nexus","level":40,"time":"2026-03-12T21:24:36.52599374Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"/home/dap/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:801","latency_us":59998957}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: 3083cb8c-d125-4c8e-8a04-89f7638e3e1d (physical_disk),\n        time_created: 2026-03-12T21:23:40.640785Z,\n        time_modified: 2026-03-12T21:23:40.640785Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g1_4\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: 09d5c82d-07ee-4f9a-9985-e2609f9a2212 (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:09.401890327Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: 51e8c233-72ea-4f5a-8488-3b2aaf1564d2 (physical_disk),\n        time_created: 2026-03-12T21:23:40.640785Z,\n        time_modified: 2026-03-12T21:23:40.640785Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g1_3\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: 09d5c82d-07ee-4f9a-9985-e2609f9a2212 (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:11.697642924Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: c9cb8b81-ff8b-49cb-8f85-93c45091358e (physical_disk),\n        time_created: 2026-03-12T21:23:40.640785Z,\n        time_modified: 2026-03-12T21:23:40.640785Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g1_1\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: 09d5c82d-07ee-4f9a-9985-e2609f9a2212 (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:12.148888282Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: e2164791-b4a4-4f9d-ac51-7474d0532a90 (physical_disk),\n        time_created: 2026-03-12T21:23:40.640785Z,\n        time_modified: 2026-03-12T21:23:40.640785Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g1_0\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: 09d5c82d-07ee-4f9a-9985-e2609f9a2212 (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:12.262539048Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: 49f6ea6a-48fa-4652-92de-bbf57d0f1608 (physical_disk),\n        time_created: 2026-03-12T21:23:40.640785Z,\n        time_modified: 2026-03-12T21:23:40.640785Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g3_2\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: 9f81b05f-7e75-42f0-af59-0681d8278766 (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:12.736930561Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: 50d481b1-8134-407e-b7e8-2859ddf14045 (physical_disk),\n        time_created: 2026-03-12T21:23:40.640785Z,\n        time_modified: 2026-03-12T21:23:40.640785Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g3_3\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: 9f81b05f-7e75-42f0-af59-0681d8278766 (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:12.753940309Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: 5bcae8dc-99bf-442d-9301-1058fb9a7364 (physical_disk),\n        time_created: 2026-03-12T21:23:40.640785Z,\n        time_modified: 2026-03-12T21:23:40.640785Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g3_0\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: 9f81b05f-7e75-42f0-af59-0681d8278766 (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:12.77231725Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: b8faaa83-7b7c-47df-a1c1-e7a6e9732fdd (physical_disk),\n        time_created: 2026-03-12T21:23:40.640785Z,\n        time_modified: 2026-03-12T21:23:40.640785Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g3_1\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: 9f81b05f-7e75-42f0-af59-0681d8278766 (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:12.78936376Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"physical disk upsert in handoff: PhysicalDisk {\n    identity: PhysicalDiskIdentity {\n        id: d06ae288-fd78-4ff8-81e1-2cb80115aee1 (physical_disk),\n        time_created: 2026-03-12T21:23:40.640785Z,\n        time_modified: 2026-03-12T21:23:40.640785Z,\n    },\n    time_deleted: None,\n    rcgen: Generation(\n        Generation(\n            1,\n        ),\n    ),\n    vendor: \"synthetic-vendor\",\n    serial: \"synthetic-serial-g3_4\",\n    model: \"synthetic-model-U2\",\n    variant: U2,\n    sled_id: 9f81b05f-7e75-42f0-af59-0681d8278766 (sled),\n    disk_policy: InService,\n    disk_state: Active,\n}","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:12.807560777Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:878"}
{"msg":"Inserted physical disks","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:12.824242967Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:890"}
{"msg":"Inserted zpools","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:14.193743524Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:902"}
{"msg":"Inserted datasets","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:14.381028274Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:931"}
{"msg":"Populated DNS tables for internal DNS","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:14.989104181Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:942"}
{"msg":"Populated DNS tables for external DNS","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:15.779850246Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:951"}
{"msg":"Created recovery silo","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:16.311391809Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:464"}
{"msg":"Created recovery user","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:16.317292517Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:488"}
{"msg":"Created recovery user's password","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:16.32372692Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:502"}
{"msg":"Generated role assignment queries","v":0,"name":"nexus","level":20,"time":"2026-03-12T21:25:16.338726121Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf"}
{"msg":"Granted Silo privileges","v":0,"name":"nexus","level":30,"time":"2026-03-12T21:25:16.353186278Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"user_builtin_id":"001de000-05e4-4000-8000-000000000002","type":"user_builtin","authenticated":true,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"nexus/db-queries/src/db/datastore/rack.rs:526"}
{"msg":"request completed after handler was already cancelled","v":0,"name":"nexus","level":40,"time":"2026-03-12T21:25:17.724329Z","hostname":"oxz_nexus_2a99b4ab-2066-4400-bb31-223f54df12bf","pid":13958,"uri":"/racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete","method":"PUT","req_id":"76d261d4-46a7-491b-b15e-93895ec4bfb0","remote_addr":"[fd00:17:1:d01::1]:60831","local_addr":"[fd00:17:1:d02::5]:12232","component":"dropshot_lockstep","name":"2a99b4ab-2066-4400-bb31-223f54df12bf","file":"/home/dap/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:948","error_message_external":"Internal Server Error","error_message_internal":"database error (kind = SerializationFailure): restart transaction: TransactionRetryWithProtoRefreshError: WriteTooOldError: write for key /Table/112/1/\"\\x13\\xf5\\x133\\xf4\\xfeK\\x17\\x96\\x92\\b\\xc2Ը\\xbeY\"/0 at timestamp 1773350654.862186262,0 too old; wrote at 1773350704.196933029,1: \"sql txn\" meta={id=cf25a7d5 key=/Table/220/1/\"\\x8c*\\\"3\\x85I@1\\x92\\x86\\xb0\\xf2A\\xa6\\xef6\"/0 pri=0.01027388 epo=0 ts=1773350713.407042155,2 min=1773350654.862186262,0 seq=681} lock=true stat=PENDING rts=1773350654.862186262,0 wto=false gul=1773350655.362186262,0\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html\n","response_code":500}

The client had timed it out, based on these messages:

21:24:36.525Z WARN 2a99b4ab-2066-4400-bb31-223f54df12bf (dropshot_lockstep): request handling cancelled (client disconnected)
    file = /home/dap/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:801
    latency_us = 59998957
    local_addr = [fd00:17:1:d02::5]:12232
    method = PUT
    remote_addr = [fd00:17:1:d01::1]:60831
    req_id = 76d261d4-46a7-491b-b15e-93895ec4bfb0
    uri = /racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete
21:25:17.724Z WARN 2a99b4ab-2066-4400-bb31-223f54df12bf (dropshot_lockstep): request completed after handler was already cancelled
    error_message_external = Internal Server Error
    error_message_internal = database error (kind = SerializationFailure): restart transaction: TransactionRetryWithProtoRefreshError: WriteTooOldError: write for key /Table/112/1/"\\x13\\xf5\\x133\\xf4\\xfeK\\x17\\x96\\x92\\b\\xc2\u{538}\\xbeY"/0 at timestamp 1773350654.862186262,0 too old; wrote at 1773350704.196933029,1: "sql txn" meta={id=cf25a7d5 key=/Table/220/1/"\\x8c*\\"3\\x85I@1\\x92\\x86\\xb0\\xf2A\\xa6\\xef6"/0 pri=0.01027388 epo=0 ts=1773350713.407042155,2 min=1773350654.862186262,0 seq=681} lock=true stat=PENDING rts=1773350654.862186262,0 wto=false gul=1773350655.362186262,0\nHINT: See: https://www.cockroachlabs.com/docs/v22.1/transaction-retry-error-reference.html\n
    file = /home/dap/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.7/src/server.rs:948
    local_addr = [fd00:17:1:d02::5]:12232
    method = PUT
    remote_addr = [fd00:17:1:d01::1]:60831
    req_id = 76d261d4-46a7-491b-b15e-93895ec4bfb0
    response_code = 500
    uri = /racks/13f51333-f4fe-4b17-9692-08c2d4b8be59/initialization-complete
root@g1:~#

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the handoff path at sled-agent/src/rack_setup/service.rs:1061 and the rack initialization handler in nexus/src/app/rack.rs, especially the logged lines around 309, 329, 339, and 365. Compare the failed handoffs with the Nexus initialization timeline and the datastore locations in nexus/db-queries/src/db/datastore/bgp.rs, deployment.rs, and rack.rs. Done means a4x2 rack setup no longer produces the reported 404 or communication failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.