CockroachDB lost quorum on critical ranges during racklette update

Separating the CockroachDB behavior out of #10640. Summarizing the sequence of steps from that issue:

1. Clean slate mupdate done with fridge to a3fee0ec59cc2beca911d13e3dc5a58dffbed8db last week, just a normal install process
2. Madrid was used normally for days to run tests where the rack was fully loaded (128 vCPUs and 768GB RAM on each sled)
3. An online update was performed to 21.0.0-0.ci+git8b0886dbd02 (tip of main at the time) using the PA automated pytest scripts.
4. Test hung due to the finish-recovery step not being done (necessary on a racklette):
5. The update was canceled, and oxide system update recovery-finish --system-version "20.0.0-0.ci+gita3fee0ec59c" was run to clean that up. Everything back online, working with no issues.
6. A new online update was done to the tip of main: 21.0.0-0.ci+git76aa5fc0607
   ...
   The following morning when I checked the status I found sled 14 down and unable to continue sled-agent after maintenance clear.

In that issue, @jgallagher observed:

Something about this update caused cockroach to become quite unhealthy. Currently with sled 14 out of action there are still 3 nodes up, and the queries I've tried by hand (not very many) work in the crdb shell...But I can't use \d to list the schema...and the still-up Nexus logs are full of failures to do auth checks on db queries.

I verified that:

• 2 of the 5 CockroachDB nodes are on sled 14, where sled agent went into maintenance and there are no control plane zones running (including the two CockroachDB nodes). I confirmed these two nodes were on sled 14 based on the sled agent's ledger.
• On some systems (e.g., oxz_cockroachdb_b5185d97-2221-4d2b-99c9-11023a3ef748 on sled 16), I'm not able to connect to Cockroach at all:

root@oxz_cockroachdb_b5185d97:~# time cockroach sql --url postgresql://[fd2a:6309:82b3:103::3]:32221/?sslmode=disable
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB OSS v22.1.22-64-g86fdbfca06 (x86_64-pc-solaris2.11, built 2026/03/18 01:49:10, go1.22.11) (same version as client)
# Cluster ID: a8871125-8b93-4ea6-8d8d-d2a198d9f64b
#
# Enter \? for a brief introduction.
#
warning: error retrieving the database name: pq: replica unavailable: (n1,s1):1 unable to serve request to r26:/NamespaceTable/{30-Max} [(n1,s1):1, (n5,s5):2, (n4,s4):4, next=6, gen=14]: lost quorum (down: (n5,s5):2,(n4,s4):4); closed timestamp: 1781768163.604528851,0 (2026-06-18 07:36:03); raft status: {"id":"1","term":81,"vote":"1","commit":2814,"lead":"1","raftState":"StateLeader","applied":2814,"progress":{"1":{"match":298784,"next":298785,"state":"StateReplicate"},"2":{"match":2814,"next":2815,"state":"StateProbe"},"4":{"match":2814,"next":2815,"state":"StateProbe"}},"leadtransferee":"0"}: operation "probe" timed out after 1m0.002s (given timeout 1m0s): result is ambiguous: after 60.00s of attempting command: context deadline exceeded

I was able to do some stuff from a different node:

root@oxz_cockroachdb_e176b8cf:/data/logs# cockroach node status --url postgres://[fd2a:6309:82b3:103::3]:32221/omicron?sslmode=disable
warning: --url specifies database "omicron", but command "status" does not accept a database name - database name ignored
  id |            address            |          sql_address          |          build          |         started_at         |         updated_at         | locality | is_available | is_live
-----+-------------------------------+-------------------------------+-------------------------+----------------------------+----------------------------+----------+--------------+----------
   1 | [fd2a:6309:82b3:101::3]:32221 | [fd2a:6309:82b3:101::3]:32221 | v22.1.22-64-g86fdbfca06 | 2026-06-18 07:25:09.448538 | 2026-06-18 18:05:36.972416 |          | true         | true
   2 | [fd2a:6309:82b3:103::3]:32221 | [fd2a:6309:82b3:103::3]:32221 | v22.1.22-64-g86fdbfca06 | 2026-06-18 06:38:51.033207 | 2026-06-18 18:05:39.711028 |          | true         | true
   3 | [fd2a:6309:82b3:102::3]:32221 | [fd2a:6309:82b3:102::3]:32221 | v22.1.22-64-g86fdbfca06 | 2026-06-18 07:36:03.677898 | 2026-06-18 18:05:39.036302 |          | true         | true
   4 | NULL                          | NULL                          | NULL                    | NULL                       | 2026-06-18 07:36:14.557497 | NULL     | false        | false
   5 | NULL                          | NULL                          | NULL                    | NULL                       | 2026-06-18 07:36:21.250976 | NULL     | false        | false
(5 rows)

root@oxz_cockroachdb_e176b8cf:/data/logs# cockroach node status --url postgres://[fd2a:6309:82b3:103::3]:32221/omicron?sslmode=disable --all
warning: --url specifies database "omicron", but command "status" does not accept a database name - database name ignored
  id |            address            |          sql_address          |          build          |         started_at         |         updated_at         | locality | is_available | is_live | replicas_leaders | replicas_leaseholders | ranges | ranges_unavailable | ranges_underreplicated | live_bytes | key_bytes  | value_bytes | intent_bytes | system_bytes | gossiped_replicas | is_decommissioning | membership | is_draining
-----+-------------------------------+-------------------------------+-------------------------+----------------------------+----------------------------+----------+--------------+---------+------------------+-----------------------+--------+--------------------+------------------------+------------+------------+-------------+--------------+--------------+-------------------+--------------------+------------+--------------
   1 | [fd2a:6309:82b3:101::3]:32221 | [fd2a:6309:82b3:101::3]:32221 | v22.1.22-64-g86fdbfca06 | 2026-06-18 07:25:09.448538 | 2026-06-18 18:05:41.473043 |          |         true |    true |               38 |                    37 |    267 |                  2 |                     39 | 3450308080 | 1031311635 |  4853469292 |         3135 |       238982 |        267        |              false |   active   |    false
   2 | [fd2a:6309:82b3:103::3]:32221 | [fd2a:6309:82b3:103::3]:32221 | v22.1.22-64-g86fdbfca06 | 2026-06-18 06:38:51.033207 | 2026-06-18 18:05:44.210595 |          |         true |    true |              209 |                   209 |    266 |                  4 |                    210 | 3450329536 | 1031292344 |  4853510074 |         3135 |       237716 |        266        |              false |   active   |    false
   3 | [fd2a:6309:82b3:102::3]:32221 | [fd2a:6309:82b3:102::3]:32221 | v22.1.22-64-g86fdbfca06 | 2026-06-18 07:36:03.677898 | 2026-06-18 18:05:43.533925 |          |         true |    true |               19 |                    19 |    261 |                  0 |                     19 | 3450226237 | 1031286016 |  4853412855 |         3135 |       232719 |        261        |              false |   active   |    false
   4 | NULL                          | NULL                          | NULL                    | NULL                       | 2026-06-18 07:36:14.557497 |     NULL |        false |   false |               46 |                    46 |    267 |                  0 |                     46 | 3109522108 | 1023180047 |  4505243448 |         3135 |       239272 |       NULL        |              false |   active   |    false
   5 | NULL                          | NULL                          | NULL                    | NULL                       | 2026-06-18 07:36:21.250976 |     NULL |        false |   false |                1 |                     1 |    268 |                  0 |                      1 | 3109428710 | 1023178845 |  4505144394 |         3135 |       240034 |       NULL        |              false |   active   |    false
(5 rows)

This looks like:

• nodes 1-3 are online and fine
• nodes 4-5 are unavailable and haven't been seen since at least this node started

Sled 14 has nodes 4 and 5. (I forgot how I confirmed this, but it's consistent with those being offline; with some messages in other nodes about their circuit breakers to n4 and n5 being tripped; and it should be verifiable by looking at the IP addresses in the ledgers and logs.)

Start by correlating the racklette update and recovery-finish sequence with sled-agent maintenance, the sled-agent ledgers, and CockroachDB logs and node-status output. Confirm why the two CockroachDB nodes on sled 14 became unavailable and identify the change needed to prevent critical ranges from losing quorum during an update.