Hubris internal and external timeouts and retries

This issue is a general call for improved testing and evaluation of timeouts and retries in our protocols.
It may be worth exploring protocol changes that judge failures by "failure to progress" instead of a simple timeout. We might also want to use increasing timeouts per retry on the assumption that time needed to satisfy the request could be the problem rather than device readiness or message loss.

While doing testing against sp-emu, a slower emulated environment, some of the timeouts and retries used in MGS needed to be adjusted in order for tests to exercise the code.

The interaction of timeouts and retries in the stack create complex real-time requirements on Hubris and the tasks in any instance of an SP+RoT. To the extent possible, each of the timeout and retry code paths should be tested with an eye kept on changes from release to release.
Ideally, we could move many of these cases to an implementation that was more tolerant of variations in timing while providing the information needed to address performance outliers in the field.

---

Prompt to generate survey:

Regenerating the SP timeouts survey

Paste the prompt below into a fresh Claude Code session to rebuild
sp-timeouts-and-retries.md against current
sources. Set REPOS to the directory holding the checkouts first.

Survey every timeout and retry on the API surfaces the SP presents, plus the
ones internal to the SP/RoT complex. Write the result to `sp-timeouts-and-retries.md`.

Repos under $REPOS, all plain checkouts; `git fetch --all` in each first:
  omicron                     (origin/main)
  management-gateway-service  (origin/main, but omicron pins a rev in
                               Cargo.toml: cite the pinned rev, not the tip)
  hubris                      (origin/master)

Scope, one section each:
  - Control plane to SP over Ethernet/UDP: nexus/mgs-updates,
    clients/gateway-client, gateway/ and smf/mgs config, gateway-sp-comms
    (retry config, discovery, ereport, busy backoff), and the SP side in hubris
    task/control-plane-agent, task/net, task/udprpc, task/dump-agent
  - The firmware update flow layered on that surface (SP, RoT, RoT bootloader,
    host phase 1)
  - Host to SP over IPCC/uart: omicron ipcc crate and its callers, hubris
    task/host-sp-comms, and the host phase 2 fetch in control-plane-agent
  - Serial console
  - SP to RoT: drv/stm32h7-sprot-server and drv/lpc55-sprot-server
  - RoT to SP: drv/lpc55-swd

Per entry: layer, constant or mechanism, value, citation, failure mode. Record
absences too (a surface with no timeout is a finding, not a gap), and note
where a source comment calls a value arbitrary, untuned, or a guess.

Cite as $REPO@$SHORTSHA:$FILE:$LINE with $FILE relative to the repo root.
Verify every line number with `git show <ref>:$FILE`, not the working tree.
Hubris timer ticks are milliseconds; say so.

Close with notes on how the layers interact: nested budgets where an inner
retry outlives an outer poll interval, loops with no deadline of their own that
depend on a caller's timeout, and attempt counts that multiply across layers.

Follow the repo and global writing rules. No agents or workflows; grep directly.

---

Every timeout and retry on the API surfaces the SP presents, plus the ones
internal to the SP and RoT complex, as a reference for reasoning about failures
in the protocols that carry these requests.

The logic spans three repos. Hubris implements the SP side of each surface.
gateway-sp-comms, inside MGS, implements the UDP retry policy toward the SP.
Nexus owns the update state machine and its deadlines. The layers stack: a
single Nexus request to MGS sits on top of a full MGS retry sequence to the SP,
which in turn may sit on top of an SP-to-RoT sprot exchange with its own
attempt count.

Line numbers were verified with git show against the mainline branch of each
repo, or against the pinned revision where one repo pins another.

• omicron@df990b0578 (origin/main)
• mgs@6c0aca2, pinned by omicron:Cargo.toml:551-553. MGS origin/main is
  438cd18; the values there are identical and only line numbers differ
• hubris@0d1ba0453 (origin/master)

Hubris timer ticks are milliseconds.

Control plane to SP: Ethernet, UDP

The management network surface. MGS speaks UDP to control-plane-agent on the
SP; Nexus speaks HTTP to MGS.

Layer	Constant / mechanism	Value	Repo@commit:file:line	Failure mode
gateway-sp-comms	SpRetryConfig	n/a	mgs@6c0aca2:gateway-sp-comms/src/single_sp.rs:148	per_attempt_timeout :150, max_attempts_reset :159, max_attempts_general :166. The whole UDP retry policy
gateway-sp-comms	attempt-cap selection	n/a	mgs@6c0aca2:gateway-sp-comms/src/single_sp.rs:2527	Exhaustion returns ExhaustedNumAttempts at :2556; resend interval applied at :2584
gateway-sp-comms	sp_busy_policy()	20ms to 1s, max_elapsed_time: None	mgs@6c0aca2:gateway-sp-comms/src/single_sp.rs:2761	Unbounded at :2770: an SP that answers Busy forever is retried forever; only the caller's timeout ends it
gateway-sp-comms	SLEEP_BETWEEN_DISCOVERY_RETRY	1s	mgs@6c0aca2:gateway-sp-comms/src/single_sp.rs:2338	Initial discovery loops forever; a misconfigured interface never succeeds
gateway-sp-comms	DISCOVERY_INTERVAL_IDLE	60s	mgs@6c0aca2:gateway-sp-comms/src/single_sp.rs:104	Background re-discovery cadence once idle
gateway-sp-comms	recv error backoff	1s	mgs@6c0aca2:gateway-sp-comms/src/shared_socket.rs:549	Socket receive errors retry unbounded
gateway-sp-comms, ereport	reuses max_attempts_general	5 attempts	mgs@6c0aca2:gateway-sp-comms/src/ereport.rs:221	Per-attempt tokio::time::timeout at :231-232; gives up at :288
MGS config, deployed	max_attempts_general	5	omicron@df990b0578:smf/mgs/config.toml:22	5 x 2s = 10s ceiling for ordinary SP requests
MGS config, deployed	max_attempts_reset	30	omicron@df990b0578:smf/mgs/config.toml:26	30 x 2s = 60s ceiling for reset requests
MGS config, deployed	per_attempt_timeout_millis	2000	omicron@df990b0578:smf/mgs/config.toml:29	UDP resend interval to the SP
MGS config, plumbing	RetryConfig -> SpRetryConfig	n/a	omicron@df990b0578:gateway/src/management_switch.rs:63	Fields at :63-65, conversion at :68
MGS config, test	same three keys	3 / 10 / 1000ms	omicron@df990b0578:gateway-test-utils/configs/config.test.toml:27	Also :31, :34. Tests run ~6x tighter than production
MGS metrics	SP_POLL_INTERVAL	1s	omicron@df990b0578:gateway/src/metrics.rs:138	Sensor polling cadence per SP; OXIMETER_COLLECTION_INTERVAL 10s at :143
SP, control-plane-agent	MAX_NET_RESTART_RETRIES	3	hubris@0d1ba0453:task/control-plane-agent/src/main.rs:529	Send retries when the net task restarts under it; then yields until the next notification
SP, udprpc	retry on net restart	unbounded	hubris@0d1ba0453:task/udprpc/src/main.rs:154	Also :175. Immediate resend, no attempt cap
SP, dump-agent	no retry	n/a	hubris@0d1ba0453:task/dump-agent/src/udp.rs:130	When the dumper dies the SP drops the request; the host is expected to retry
SP, net	SOCKET_QUEUE_FULL_TIMEOUT_MS	500ms	hubris@0d1ba0453:task/net/src/server.rs:791	A socket whose queue stays full this long rolls to QueueFullTimeout at :804, checked at :427
SP, net	WAKE_INTERVAL	500ms typical, 5s on some boards	hubris@0d1ba0453:task/net/src/bsp/gimlet_bcdef.rs:29	Default is None at bsp_support.rs:67; per-BSP link polling cadence

Control plane to SP: firmware update flow

Layered on the UDP surface above. Nexus drives the state machine; every request
it issues inherits the MGS retry policy.

Layer	Constant / mechanism	Value	Repo@commit:file:line	Failure mode
Nexus driver	PROGRESS_TIMEOUT	600s	omicron@df990b0578:nexus/mgs-updates/src/driver_update.rs:67	Checked at :549, also driver.rs:347. No status change for 10 min aborts the attempt
Nexus driver	RESET_DELAY_INTERVAL	10s	omicron@df990b0578:nexus/mgs-updates/src/driver_update.rs:70	Sleep at :442 between failed device resets; retries until PROGRESS_TIMEOUT
Nexus driver	PROGRESS_POLL_INTERVAL	10s	omicron@df990b0578:nexus/mgs-updates/src/driver_update.rs:73	Sleep at :750 while waiting for the device on the new version
Nexus driver	STATUS_POLL_INTERVAL	3s	omicron@df990b0578:nexus/mgs-updates/src/common_sp_update.rs:31	Sleep at driver_update.rs:587 during upload/prepare
Nexus driver	DEFAULT_RETRY_TIMEOUT	60s	omicron@df990b0578:nexus/mgs-updates/src/driver_update.rs:76	Re-exported lib.rs:31, wired at nexus/src/app/mod.rs:133,543
Nexus driver	retry timer arming	n/a	omicron@df990b0578:nexus/mgs-updates/src/driver.rs:411	delayq.insert at :413 runs after every attempt regardless of outcome; fires at :445
Nexus, RoT	WAIT_FOR_BOOT_INFO_TIMEOUT	120s	omicron@df990b0578:nexus/mgs-updates/src/rot_updater.rs:30	Passed at :259; elapsed checked at :302 and :324
Nexus, RoT	WAIT_FOR_BOOT_INFO_INTERVAL	10s	omicron@df990b0578:nexus/mgs-updates/src/rot_updater.rs:32	"will retry" logged at :315, :337
Nexus, RoT bootloader	reuses the RoT constant	120s	omicron@df990b0578:nexus/mgs-updates/src/rot_bootloader_updater.rs:15	Applied at :196 and :275, around both bootloader resets
Nexus, host phase 1	PHASE_1_HASHING_TIMEOUT	60s	omicron@df990b0578:nexus/mgs-updates/src/host_phase1_updater.rs:159	Passed to the client hash poll at :323
Nexus, MGS selection	RetryableMgsError	n/a	omicron@df990b0578:nexus/mgs-updates/src/mgs_clients.rs:28	try_all_serially at :107 walks to the next MGS on a retryable error, logging at :132
Nexus, MGS selection	phase-1 hash timeout is non-retryable	n/a	omicron@df990b0578:nexus/mgs-updates/src/mgs_clients.rs:60	Returns immediately: the SP, not MGS, is the slow party
gateway-client	SLEEP_BETWEEN_POLLS	1s	omicron@df990b0578:clients/gateway-client/src/lib.rs:146	In host_phase_1_flash_hash_calculate_with_timeout (:116), sleep at :192
gateway-sp-comms	POLL_UPDATE_STATUS_INTERVAL	2s	mgs@6c0aca2:gateway-sp-comms/src/single_sp/update.rs:593	Sleep at :615 in poll_until_update_prep_complete; no deadline in this loop, it relies on the caller
gateway-sp-comms	reset_watchdog_timeout_ms()	1.5x reset ceiling	mgs@6c0aca2:gateway-sp-comms/src/single_sp.rs:170	SP watchdog fires after the MGS timeout, so MGS gives up first

Host to SP: IPCC over the uart

The host speaks to host-sp-comms over a uart. The omicron side is a thin
ioctl wrapper with no deadline of its own.

Layer	Constant / mechanism	Value	Repo@commit:file:line	Failure mode
Host, ipcc crate	none	n/a	omicron@df990b0578:ipcc/src/lib.rs	319 lines, zero matches for timeout, retry, or sleep. Ipcc::new at :162, installinator_image_id at :168, its key_lookup call at :174. The ioctl blocks on the driver; the deadline, if any, lives in libipcc (Cargo.toml:619) and the illumos driver, neither in this tree
Host, installinator	none	n/a	omicron@df990b0578:installinator/src/artifact.rs:46	A failed IPCC lookup aborts the run; no retry around it
SP, host-sp-comms	UART_ZERO_DELAY	200ms	hubris@0d1ba0453:task/host-sp-comms/src/main.rs:92	Cadence for sending 0x00 framing bytes while idle, so the host can resynchronize
SP, host-sp-comms	A2_REBOOT_DELAY	5000ms	hubris@0d1ba0453:task/host-sp-comms/src/main.rs:87	Fixed wait for rails to decay before returning to A0. The comment calls the value a guess
SP, host-sp-comms	A1 sleep-and-retry	n/a	hubris@0d1ba0453:task/host-sp-comms/src/main.rs:461	Power-off retries against a transitory A1 state with no attempt cap
SP, host phase 2	DELAY_TRY_OTHER_MGS	500ms	hubris@0d1ba0453:task/control-plane-agent/src/mgs_compute_sled/host_phase2.rs:52	No answer from the first MGS in this window flips to the other port
SP, host phase 2	DELAY_RETRY	1000ms	hubris@0d1ba0453:task/control-plane-agent/src/mgs_compute_sled/host_phase2.rs:53	Window for the second MGS before starting a new round
SP, host phase 2	MAX_ATTEMPTS	6	hubris@0d1ba0453:task/control-plane-agent/src/mgs_compute_sled/host_phase2.rs:54	Checked at :151. The SP then reports failure to the host, which decides whether to ask again

Serial console

Layer	Constant / mechanism	Value	Repo@commit:file:line	Failure mode
Protocol	SERIAL_CONSOLE_IDLE_TIMEOUT	20s	mgs@6c0aca2:gateway-messages/src/lib.rs:33	Shared constant, enforced on the SP
SP, control-plane-agent	idle-client eviction	20s	hubris@0d1ba0453:task/control-plane-agent/src/mgs_compute_sled.rs:292	A client whose last keepalive is older than the timeout is detached and its rx data dropped
SP, control-plane-agent	SERIAL_CONSOLE_FLUSH_TIMEOUT_MILLIS	500ms	hubris@0d1ba0453:task/control-plane-agent/src/mgs_compute_sled.rs:90	Deadline set at :1481; bounds how long host output waits for a full packet
MGS client	keepalive cadence	5s	mgs@6c0aca2:faux-mgs/src/usart.rs:197	SERIAL_CONSOLE_IDLE_TIMEOUT / 4, so three keepalives can be lost before eviction

SP to RoT: sprot over SPI

Internal to the SP and RoT complex, but every control-plane request that
touches the RoT crosses it, so its attempt counts multiply with the MGS ones.

Layer	Constant / mechanism	Value	Repo@commit:file:line	Failure mode
SP, sprot server	do_send_recv_retries	n/a	hubris@0d1ba0453:drv/stm32h7-sprot-server/src/main.rs:472	Retries only errors that report as recoverable; anything else returns on the first failure
SP, sprot server	RETRY_TIMEOUT	5 ticks	hubris@0d1ba0453:drv/stm32h7-sprot-server/src/main.rs:77	Sleep between attempts at :547
SP, sprot server	DEFAULT_ATTEMPTS	3	hubris@0d1ba0453:drv/stm32h7-sprot-server/src/main.rs:82	Covers the fail, pulse-CSn, retry sequence
SP, sprot server	TIMEOUT_QUICK	5 ticks	hubris@0d1ba0453:drv/stm32h7-sprot-server/src/main.rs:80	Status and other short messages
SP, sprot server	TIMEOUT_MEDIUM	50 ticks	hubris@0d1ba0453:drv/stm32h7-sprot-server/src/main.rs:84	Update block writes, paired with MAX_UPDATE_ATTEMPTS
SP, sprot server	TIMEOUT_LONG	200 ticks	hubris@0d1ba0453:drv/stm32h7-sprot-server/src/main.rs:86	A source comment flags this as untuned, and notes a retried finish_image_update is unhandled
SP, sprot server	MAX_UPDATE_ATTEMPTS	3	hubris@0d1ba0453:drv/stm32h7-sprot-server/src/main.rs:97	Attempt cap for update operations
SP, sprot server	DUMP_TIMEOUT	1000 ticks	hubris@0d1ba0453:drv/stm32h7-sprot-server/src/main.rs:106	The RoT halts the SP to take a dump, so SP time does not advance. 5ms was found to be too short
SP, sprot server	PART2_DELAY	2 ticks	hubris@0d1ba0453:drv/stm32h7-sprot-server/src/main.rs:95	Gap after the FIFO-sized first chunk, giving the RoT time to service its interrupt. PART1_DELAY is 0 at :90
SP, sprot server	wait_rot_irq	caller's timeout	hubris@0d1ba0453:drv/stm32h7-sprot-server/src/main.rs:380	Arms a timer at :399 and races it against the ROT_IRQ notification, preferring the IRQ when both are posted
RoT, sprot server	none	n/a	hubris@0d1ba0453:drv/lpc55-sprot-server/src/main.rs	No timeout or retry on the RoT side. It is interrupt-driven and answers when addressed; all recovery is the SP's

The block comment at drv/stm32h7-sprot-server/src/main.rs:72-75 states these
timeouts are arbitrary and not yet configurable.

RoT to SP: swd

The RoT drives the SP's debug port to halt, measure, and reset it.

Layer	Constant / mechanism	Value	Repo@commit:file:line	Failure mode
RoT, swd	WAIT_FOR_HALT_MS	500ms	hubris@0d1ba0453:drv/lpc55-swd/src/main.rs:279	wait_for_sp_halt at :1624 polls every 1ms and returns SpCtrlError::Timeout at :1653
RoT, swd	RETRY_LIMIT	10	hubris@0d1ba0453:drv/lpc55-swd/src/main.rs:1486	Polls the register-ready bit at 1ms intervals, then reports a fault

A held physical reset button keeps SP_RESET asserted, so the SP never halts and
wait_for_sp_halt times out. The code reads the reset line on timeout to make
that case distinguishable in the ringbuf.

Notes

The deployed 60s reset ceiling (30 attempts x 2s) sits under Nexus's 10s
RESET_DELAY_INTERVAL. Nexus issues a new reset request roughly every 10s while
MGS is still spending up to 60s retrying the previous one. Reset requests queue
rather than serialize cleanly.

Three loops in gateway-sp-comms carry no deadline of their own: sp_busy_policy
(max_elapsed_time: None), the initial discovery retry, and
poll_until_update_prep_complete. Each depends on a caller-supplied timeout to
terminate. On the update path that caller is Nexus's PROGRESS_TIMEOUT, ten
minutes away.

Attempt counts multiply across layers. An RoT-touching request retried 5 times
by MGS, each attempt crossing sprot with 3 attempts, is 15 SPI exchanges for
one control-plane call. Nothing in either layer knows the other's budget.

The host phase 2 path crosses three surfaces in one request: host to SP over
the uart, SP to MGS over UDP, and MGS to the control plane. The SP's own
MAX_ATTEMPTS of 6 spans roughly 9 seconds, and on exhaustion it returns a
failure the host is free to reissue immediately.

Start with sp-timeouts-and-retries.md and trace the cited entry points across omicron, management-gateway-service, and hubris, including gateway-sp-comms, control-plane-agent, and the SP/RoT communication tasks. Run the existing timeout and retry tests, then define focused coverage for the listed paths and timing interactions; done should include reproducible tests and an agreed approach for failures that currently depend on nested or unbounded retries.