end-to-end-tests (helios-deploy job) flaky due to omicron#3082?

We've seen several failures of the end-to-end job in CI lately, including on "main". I looked at this one:
https://github.com/oxidecomputer/omicron/runs/13735502013

It timed out trying to reach Nexus after 5 minutes:
https://buildomat.eng.oxide.computer/wg/0/details/01H17JHYNCVECGSR5G7Y092DRW/g1voiBAdJhCjuG8OrVz0o6RHzU78LGcP6TUHReJYAfE52AwE/01H17JK3X9NZ7MYVV8X18JGCJ2?ts=all#S609

Well, is the hostname in external DNS? The external-dns server log shows that external DNS came up, but never received a request to update configuration to advertise any DNS names.

Nexus normally does this. What's it doing? I looked at its log, which ends with:

2023-05-24T20:25:21.781Z	INFO	nexus (dropshot_internal): request completed
    error_message_external = not found: zpool with id "f4b4dc87-ab46-49fb-a4b4-d361ae214c03"
    error_message_internal = not found: zpool with id "f4b4dc87-ab46-49fb-a4b4-d361ae214c03"
    local_addr = [fd00:1122:3344:101::4]:12221
    method = PUT
    remote_addr = [fd00:1122:3344:101::1]:40469
    req_id = 9c47abd2-25d4-443f-8f3c-c7f8ce56fa56
    response_code = 404
    uri = /racks/327b9e22-d506-4e4b-8546-82149102718d/initialization-complete

That sounds like #3082, so I looked at all the requests it's served:

$ curl -s https://buildomat.eng.oxide.computer/wg/0/artefact/01H17JHYNCVECGSR5G7Y092DRW/g1voiBAdJhCjuG8OrVz0o6RHzU78LGcP6TUHReJYAfE52AwE/01H17JK3X9NZ7MYVV8X18JGCJ2/01H17PY65HD37339KHZG5GPZVH/oxide-nexus:default.log | grep '"request completed"' | json -ga time response_code uri error_message_external
2023-05-24T13:24:34.897678984-07:00 204 /metrics/collectors 
2023-05-24T13:24:35.400173293-07:00 204 /metrics/producers 
2023-05-24T13:24:40.669917198-07:00 404 /sled-agents/6f0f865b-7d3c-40ef-a8b9-18c66c253d85/zpools/e4b4dc87-ab46-49fb-a4b4-d361ae214c03 not found: physical-disk with id "vendor = "synthetic-vendor", serial = "synthetic-serial-e4b4dc87-ab46-49fb-a4b4-d361ae214c03", model = "synthetic-model""
2023-05-24T13:24:47.227589048-07:00 204 /sled-agents/6f0f865b-7d3c-40ef-a8b9-18c66c253d85 
2023-05-24T13:24:47.230850656-07:00 204 /sled-agents/6f0f865b-7d3c-40ef-a8b9-18c66c253d85 
2023-05-24T13:24:48.425365496-07:00 200 /physical-disk 
2023-05-24T13:24:49.714372893-07:00 404 /sled-agents/6f0f865b-7d3c-40ef-a8b9-18c66c253d85/zpools/f4b4dc87-ab46-49fb-a4b4-d361ae214c03 not found: physical-disk with id "vendor = "synthetic-vendor", serial = "synthetic-serial-f4b4dc87-ab46-49fb-a4b4-d361ae214c03", model = "synthetic-model""
2023-05-24T13:24:50.452996999-07:00 200 /physical-disk 
2023-05-24T13:24:50.609667384-07:00 200 /sled-agents/6f0f865b-7d3c-40ef-a8b9-18c66c253d85/zpools/d462a7f7-b628-40fe-80ff-4e4189e2d62b 
2023-05-24T13:24:53.140425713-07:00 200 /physical-disk 
2023-05-24T13:24:55.421719266-07:00 200 /physical-disk 
2023-05-24T13:24:59.488430152-07:00 200 /sled-agents/6f0f865b-7d3c-40ef-a8b9-18c66c253d85/zpools/b462a7f7-b628-40fe-80ff-4e4189e2d62b 
2023-05-24T13:25:01.868566561-07:00 200 /sled-agents/6f0f865b-7d3c-40ef-a8b9-18c66c253d85/zpools/a462a7f7-b628-40fe-80ff-4e4189e2d62b 
2023-05-24T13:25:04.66633752-07:00 200 /physical-disk 
2023-05-24T13:25:07.50863266-07:00 200 /sled-agents/6f0f865b-7d3c-40ef-a8b9-18c66c253d85/zpools/e4b4dc87-ab46-49fb-a4b4-d361ae214c03 
2023-05-24T13:25:20.430102945-07:00 404 /racks/327b9e22-d506-4e4b-8546-82149102718d/initialization-complete not found: zpool with id "f4b4dc87-ab46-49fb-a4b4-d361ae214c03"
2023-05-24T13:25:20.615126124-07:00 404 /racks/327b9e22-d506-4e4b-8546-82149102718d/initialization-complete not found: zpool with id "f4b4dc87-ab46-49fb-a4b4-d361ae214c03"
2023-05-24T13:25:20.836145674-07:00 404 /racks/327b9e22-d506-4e4b-8546-82149102718d/initialization-complete not found: zpool with id "f4b4dc87-ab46-49fb-a4b4-d361ae214c03"
2023-05-24T13:25:21.061336425-07:00 404 /racks/327b9e22-d506-4e4b-8546-82149102718d/initialization-complete not found: zpool with id "f4b4dc87-ab46-49fb-a4b4-d361ae214c03"
2023-05-24T13:25:21.45090309-07:00 404 /racks/327b9e22-d506-4e4b-8546-82149102718d/initialization-complete not found: zpool with id "f4b4dc87-ab46-49fb-a4b4-d361ae214c03"
2023-05-24T13:25:21.781534755-07:00 404 /racks/327b9e22-d506-4e4b-8546-82149102718d/initialization-complete not found: zpool with id "f4b4dc87-ab46-49fb-a4b4-d361ae214c03"

That seems consistent with #3082. We've failed a number of rack initialization requests because they depend on zpool f4b4dc87-ab46-49fb-a4b4-d361ae214c03 that we don't know about. We got a request to tell us about that, but we rejected it because we didn't know about the disk. We did get several notifications about disks after that, and some zpools, too, but not this zpool. So maybe we just ran out of time?

I don't know why this might have started recently.

Start with the helios-deploy end-to-end run and the linked Nexus and external-dns logs, then compare the failure with issue #3082. Trace the rack initialization request and the missing zpool notifications to determine why Nexus returns 404 and external DNS receives no update. Done means the CI job completes rack initialization and no longer times out waiting for Nexus or DNS.