Co-located boundary NTP and internal DNS zones may deadlock from cold boot if the remaining internal DNS zone is down

A customer rack on R21 experienced a full rack restart and failed to start control plane services. This appears to be the result of number of unlikely (but not explicitly prevented) layout of services on sleds. This is quite similar to the scenario outlined in https://github.com/oxidecomputer/omicron/issues/9700.

Prior to the rack restart, the service layout was:

• sled 17: internal DNS
• sled 22: internal DNS and boundary NTP
• sled 24: internal DNS and boundary NTP

The restart caused sled 17 to die for unknown reasons, all other sleds booted successfully, but rack services did not come online.

We found that the two boundary NTP zones had failed to start, but all other NTP zones started up without issue. The internal_dns zones on sleds 22 and 24 were not running.

support@oxz_switch0:~$ pilot host exec -c 'zoneadm list -n | grep oxz_ntp | xargs -i svcprop -p config/boundary -z {} chrony-setup' 0-31
 4  2ECA6DJA           ok: false
 5  2HDVDNY9           ok: false
 6  2NCFFFVE           ok: false
 7  2M05T914           ok: false
 8  23K20DXR           ok: false
 9  21P072GG           ok: false
10  269WD9EJ           ok: false
11  2RCAC359           ok: false
12  2D9KMEP6           ok: false
13  2CVACED6           ok: false
14  2DT04XAP           ok: false
15  290FNC6C           ok: false
16  2TPKFTRP           ok: false
18  2239KG97           ok: false
19  27M4D010           ok: false
20  2062GRAN           ok: false
21  2RY6P8MV           ok: false
22  2F6KEGJC           ok: 
23  2G1X5CMK           ok: false
24  2V7HMGJW           ok: 
25  27J987N5           ok: false
26  27YAN9PA           ok: false
27  2AX7CM9D           ok: false

On sleds 22 and 24, we observed these errors occurring regularly:

1986-12-28 17:48:07.607Z WARN SledAgent/657 (ServiceManager) on 2F6KEGJC: Failed to look up switch zone slots
    error = Error resolving dendrite services in internal DNS: proto error: io error: No route to host (os error 148)
    file = sled-agent/early-networking/src/lib.rs:210
    requested_wait_time = 300s
    retry_after = 8.756918793s
    total_elapsed = 63842.360861546s

It seems likely that during sled-agent startup on both sleds 22 and 24 the NTP zone was ready to be started before internal DNS.

Zone startup will wait for all of the attempted zone starts to exit before proceeding, which in turn blocks ReconcilerTask::run inside do_reconciliation. NTP needs internal DNS to find the switch zone, causing the error logged above, so retries in a loop forever and its future is never joined.

This means we never exit from do_reconciliation, never attempt to iterate over the loop, and don't retry starting the internal DNS zone. This in turn prevents control plane services from starting up, as time is never synced.

Start with the cited startup paths in sled-agent/config-reconciler/src/reconciler_task/zones.rs, sled-agent/config-reconciler/src/reconciler_task.rs, and sled-agent/early-networking/src/lib.rs. Trace how boundary NTP waits on internal DNS during do_reconciliation when another DNS zone is unavailable. Done means a cold boot can retry or start the remaining internal DNS zone and bring up control-plane services without the observed deadlock.