Nexus-driven update stuck constantly restarting multiple SPs

Today we attempted a Nexus-driven update on dublin, with a target TUF repo that was custom-built to include old hubris images for all hubris targets:

• SP 1.0.46 (currently on dublin: 1.0.47)
• RoT 1.0.35 (currently on dublin: 1.0.36)
• RoT bootloader 1.3.3 (currently on dublin: 1.4.0)

The update got stuck after about a dozen blueprints:

  792 2025-10-02T16:15:56.666Z 66d2079c-f4f4-4d17-8564-7445158aab51  enabled: update Switch 0 (BRM23230002) RoT bootloader to 1.3.3
  793 2025-10-02T16:16:21.899Z 844bdbec-612b-4008-9f66-b2dfee2aa228  enabled: update Switch 0 (BRM23230002) RoT to 1.0.35
  794 2025-10-02T16:17:07.360Z 6b4f75f4-df5b-4987-9221-d3768fed0325  enabled: update Switch 0 (BRM23230002) SP to 1.0.46
  795 2025-10-02T16:18:52.107Z becbcb85-ca78-4ec9-95d7-762b08976884  enabled: update Switch 1 (BRM31230002) RoT bootloader to 1.3.3
  796 2025-10-02T16:20:09.946Z 0a5d0405-e2dd-4609-8b11-b5d3ca45afb1  enabled: update Switch 1 (BRM31230002) RoT to 1.0.35
  797 2025-10-02T16:21:28.513Z fd896ea7-cc0d-43b7-afa2-a016bf75249c  enabled: update Switch 1 (BRM31230002) SP to 1.0.46
  798 2025-10-02T16:24:14.816Z 646d4925-923b-4f27-b6d6-f0e2b990eed3  enabled: update Sled 17 (BRM23230010) RoT bootloader to 1.3.3
  799 2025-10-02T16:27:16.759Z a300eadc-f80b-4d8d-b99c-50955ef16aa3  enabled: update Sled 17 (BRM23230010) RoT to 1.0.35
  800 2025-10-02T16:30:55.436Z 1ce510f2-f84c-4afc-a0b8-b291351e58a6  enabled: update Sled 17 (BRM23230010) SP to 1.0.46
  801 2025-10-02T16:37:03.460Z e9c10564-a4f4-4233-a393-c23f677ed8f0  enabled: update Sled 16 (BRM23230018) RoT bootloader to 1.3.3
  802 2025-10-02T16:40:30.386Z 233c7f5b-7500-42f3-a427-3e7eb87bb23a  enabled: update Sled 16 (BRM23230018) RoT to 1.0.35
  803 2025-10-02T16:44:00.662Z f40d61a2-2f1b-4288-9014-5699df4287c7  enabled: update Sled 16 (BRM23230018) SP to 1.0.46
  804 2025-10-02T16:48:06.411Z 157bc022-49b1-461e-a652-478561daaef1  enabled: update Sled 17 (BRM23230010) host phase 1 to 17.0.0-0.ci+gitf2a3273a8b2
  805 2025-10-02T16:53:51.063Z 549de8da-d590-42a7-a033-c45d449354ee  enabled: update Sled 17 (BRM23230010) host phase 1 to 17.0.0-0.ci+gitf2a3273a8b2

Poking around on the system, it was very obviously unhealthy. We were only able to log into tech port 0 (sled 14), from which we saw:

• pilot host ls only showed sleds 14, 15, and 17
• pilot sp ls sometimes showed sled 16's SP and sometimes did not
• pilot sp ls -x sometimes showed both sidecars, and sometimes only showed sidecar0
• the shell would periodically freeze up for several seconds

After some digging, we discovered that the SPs for sled 16, sidecar 0, and sidecar 1 were being periodically reset by Nexus. At this point I disabled both execution and automatic planning, but the periodic SP resets continued.

I'll put debugging notes in followup comments. Related PRs/issues for specific things to investigate / fix:

• #9135
• #9136
• #9137

Start with the Nexus-driven update behavior described in the issue and reproduce the periodic SP resets while execution and automatic planning are disabled. Review the pilot host and SP observations, then follow the debugging notes in the five comments and the related issues #9135, #9136, and #9137. Done means the update no longer remains stuck restarting SPs and the affected sleds and sidecars remain visible and responsive.