cockroach-admin fails to decommission nodes that are dead and gone

@leftwo ran into this on a racklette, and with nontrivial work was able to reproduce it. The initial symptom is that blueprint execution is reporting failures to contact the cockroach-admin servers:

root@oxz_switch1:~# omdb nexus background-tasks show blueprint_executor
task: "blueprint_executor"
  configured period: every 1m
  currently executing: no
  last completed activation: iter 1019, triggered by a periodic timer firing
    started at 2025-06-25T14:28:58.713Z (57s ago) and ran for 11245ms
    target blueprint: 0e1416c0-8189-4ab6-bd20-6a5f6e34b14c
    execution:        enabled
    status:           completed (14 steps)
    warning:          at: Cleanup expunged zones: 6 errors encountered: Error: failed to decommission cockroach zone 554ddee9-60e5-4310-b895-1e09b72dee8d (node 9): failed to contact 5 admin servers
                      Error: failed to decommission cockroach zone d0d53f60-7c89-42e1-bc2c-5e71c390a6d9 (node 8): failed to contact 5 admin servers
                      Error: failed to decommission cockroach zone 07ae7159-a316-4609-9566-bac6b9e7f7b8 (node 1): failed to contact 5 admin servers
                      Error: failed to decommission cockroach zone 13f5c80c-a95d-4446-bdfe-4a9529b04a4b (node 2): failed to contact 5 admin servers
                      Error: failed to decommission cockroach zone 6242d94a-3150-4b87-bc8d-aad196c934f3 (node 5): failed to contact 5 admin servers
                      Error: failed to decommission cockroach zone f39fbd20-2e72-4ffd-a803-2247b2f5935f (node 7): failed to contact 5 admin servers
    error:            (none)

This warning is spectacularly uninformative; it comes from this bit of code where we emit the real error in a warn! log, but only report the number of servers we failed to contact in the task output: https://github.com/oxidecomputer/omicron/blob/2f8d2cd48bd8fefa437a39ad0cc022e7f87c766f/nexus/reconfigurator/execution/src/omicron_zones.rs#L262-L276

The actual errors, pulled from Nexus logs, are CSV parsing failures (!):

14:00:01.141Z WARN 0e0191b4-73f8-4d6e-bed5-bb7e65b909ba (ServerContext): failed sending decommission request (will try other servers)
    admin_url = http://[fd00:1122:3344:102::21]:32222
    background_task = blueprint_executor
    comment = sled 38c7d156-93bd-4ca7-9c45-7cddde229a64 expunged (expunged 10 disks, 49 datasets, 16 zones)
    err = Error Response: status: 500 Internal Server Error; headers: {"content-type": "application/json", "x-request-id": "b3731003-7cc5-4d61-b90a-b35c06051d05", "content-length": "2908", "date": "Wed, 25 Jun 2025 14:00:00 GMT"}; value: Error { error_code: Some("Internal"), message: "failed to parse stdout \\"id,address,sql_address,build,started_at,updated_at,locality,is_available,is_live,replicas_leaders,replicas_leaseholders,ranges,ranges_unavailable,ranges_underreplicated,live_bytes,key_bytes,value_bytes,intent_bytes,system_bytes,gossiped_replicas,is_decommissioning,membership,is_draining\\\\n1,NULL,NULL,NULL,NULL,2025-06-25 01:11:12.13184,NULL,false,false,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,true,decommissioned,false\\\\n2,NULL,NULL,NULL,NULL,2025-06-25 01:11:12.131852,NULL,false,false,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,true,decommissioned,false\\\\n3,[fd00:1122:3344:102::3]:32221,[fd00:1122:3344:102::3]:32221,v22.1.22-29-g865aff1595,2025-06-24 21:41:11.713346,2025-06-25 13:59:58.549321,,true,true,43,43,200,0,0,629380384,249251356,969836910,0,628089,200,false,active,false\\\\n4,[fd00:1122:3344:101::3]:32221,[fd00:1122:3344:101::3]:32221,v22.1.22-29-g865aff1595,2025-06-24 21:41:11.840023,2025-06-25 13:59:59.947669,,true,true,40,40,200,0,0,629516359,249251404,969972973,0,628089,200,false,active,false\\\\n5,NULL,NULL,NULL,NULL,2025-06-25 01:11:12.131862,NULL,false,false,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,true,decommissioned,false\\\\n6,[fd00:1122:3344:102::21]:32221,[fd00:1122:3344:102::21]:32221,v22.1.22-29-g865aff1595,2025-06-24 21:53:06.479128,2025-06-25 13:59:57.753587,,true,true,42,42,200,0,0,629672212,249327319,970053439,0,628089,200,false,active,false\\\\n7,NULL,NULL,NULL,NULL,2025-06-25 01:11:12.131868,NULL,false,false,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,true,decommissioned,false\\\\n8,NULL,NULL,NULL,NULL,2025-06-25 01:11:13.977313,NULL,false,false,40,40,199,0,0,284711075,44513462,342213403,0,450633,NULL,false,active,false\\\\n9,NULL,NULL,NULL,NULL,2025-06-25 01:11:13.973348,NULL,false,false,35,35,199,0,0,284755105,44513474,342257456,0,450633,NULL,false,active,false\\\\n10,[fd00:1122:3344:122::2d]:32221,[fd00:1122:3344:122::2d]:32221,v22.1.22-29-g865aff1595,2025-06-25 04:13:32.276502,2025-06-25 13:59:59.114207,,true,true,36,36,200,0,0,629380384,249251356,969836910,0,628089,200,false,active,false\\\\n11,[fd00:1122:3344:122::2c]:32221,[fd00:1122:3344:122::2c]:32221,v22.1.22-29-g865aff1595,2025-06-25 04:13:43.192807,2025-06-25 14:00:01.079924,,true,true,39,39,200,0,0,629471034,249251404,969927648,0,628089,200,false,active,false\\\\n\\", stderr \\"\\": failed to parse `cockroach node status` output: failed to parse node status row StringRecord([\\"8\\", \\"NULL\\", \\"NULL\\", \\"NULL\\", \\"NULL\\", \\"2025-06-25 01:11:13.977313\\", \\"NULL\\", \\"false\\", \\"false\\", \\"40\\", \\"40\\", \\"199\\", \\"0\\", \\"0\\", \\"284711075\\", \\"44513462\\", \\"342213403\\", \\"0\\", \\"450633\\", \\"NULL\\", \\"false\\", \\"active\\", \\"false\\"]): CSV deserialize error: record 8 (line: 9, byte: 1558): invalid socket address syntax", request_id: "b3731003-7cc5-4d61-b90a-b35c06051d05" }
    file = nexus/reconfigurator/execution/src/omicron_zones.rs:263
    sled_id = 38c7d156-93bd-4ca7-9c45-7cddde229a64
    zone_id = 554ddee9-60e5-4310-b895-1e09b72dee8d
    zone_type = cockroach_db

In #8243, we added safety checks to cockroach-admin as a workaround to avoid an apparent-race in cockroach described in #8239. The safety checks are based on the parsed output of running cockroach node status --all --format csv. If we failed to parse the output we couldn't even get to performing the safety checks. Expanding the error message and stripping the debug printing, we had this stdout:

id,address,sql_address,build,started_at,updated_at,locality,is_available,is_live,replicas_leaders,replicas_leaseholders,ranges,ranges_unavailable,ranges_underreplicated,live_bytes,key_bytes,value_bytes,intent_bytes,system_bytes,gossiped_replicas,is_decommissioning,membership,is_draining
1,NULL,NULL,NULL,NULL,2025-06-25 01:11:12.13184,NULL,false,false,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,true,decommissioned,false
2,NULL,NULL,NULL,NULL,2025-06-25 01:11:12.131852,NULL,false,false,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,true,decommissioned,false
3,[fd00:1122:3344:102::3]:32221,[fd00:1122:3344:102::3]:32221,v22.1.22-29-g865aff1595,2025-06-24 21:41:11.713346,2025-06-25 13:59:58.549321,,true,true,43,43,200,0,0,629380384,249251356,969836910,0,628089,200,false,active,false
4,[fd00:1122:3344:101::3]:32221,[fd00:1122:3344:101::3]:32221,v22.1.22-29-g865aff1595,2025-06-24 21:41:11.840023,2025-06-25 13:59:59.947669,,true,true,40,40,200,0,0,629516359,249251404,969972973,0,628089,200,false,active,false
5,NULL,NULL,NULL,NULL,2025-06-25 01:11:12.131862,NULL,false,false,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,true,decommissioned,false
6,[fd00:1122:3344:102::21]:32221,[fd00:1122:3344:102::21]:32221,v22.1.22-29-g865aff1595,2025-06-24 21:53:06.479128,2025-06-25 13:59:57.753587,,true,true,42,42,200,0,0,629672212,249327319,970053439,0,628089,200,false,active,false
7,NULL,NULL,NULL,NULL,2025-06-25 01:11:12.131868,NULL,false,false,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,true,decommissioned,false
8,NULL,NULL,NULL,NULL,2025-06-25 01:11:13.977313,NULL,false,false,40,40,199,0,0,284711075,44513462,342213403,0,450633,NULL,false,active,false
9,NULL,NULL,NULL,NULL,2025-06-25 01:11:13.973348,NULL,false,false,35,35,199,0,0,284755105,44513474,342257456,0,450633,NULL,false,active,false
10,[fd00:1122:3344:122::2d]:32221,[fd00:1122:3344:122::2d]:32221,v22.1.22-29-g865aff1595,2025-06-25 04:13:32.276502,2025-06-25 13:59:59.114207,,true,true,36,36,200,0,0,629380384,249251356,969836910,0,628089,200,false,active,false
11,[fd00:1122:3344:122::2c]:32221,[fd00:1122:3344:122::2c]:32221,v22.1.22-29-g865aff1595,2025-06-25 04:13:43.192807,2025-06-25 14:00:01.079924,,true,true,39,39,200,0,0,629471034,249251404,969927648,0,628089,200,false,active,false

which failed to parse with:

failed to parse `cockroach node status` output:
failed to parse node status row StringRecord(["8", "NULL", "NULL", "NULL", "NULL", "2025-06-25 01:11:13.977313", "NULL", "false", "false", "40", "40", "199", "0", "0", "284711075", "44513462", "342213403", "0", "450633", "NULL", "false", "active", "false"]):
CSV deserialize error: record 8 (line: 9, byte: 1558): invalid socket address syntax

The CSV parser expects all of the fields to be present for any row with a membership value other than decommissioned, but for nodes 8 and 9 in the above output, we have several NULL columns despite membership being active. It's straightforward (if ugly) to change the CSV parsing to be more flexible, but our safety check wants to ensure that the node-to-be-decommissioned has no gossiped_replicas. Nodes 8 and 9 are reporting a gossiped_replicas value of NULL. Our assumption at this point was:

• There's some longer timeout in cockroach that we didn't see while investigating #8239 or developing #8243 that causes some of an active nodes status columns to go to NULL
• Hopefully gossiped_replicas went to 0 long before it became NULL. If that's true, if we had tried to decommission it while it was still 0, presumably we would have succeeded; it's only because we're coming along much later that we're seeing this state.
• Bolstering the "it's probably okay to treat NULL as 0" argument, @jmpesp noticed that this output is produced via a left join, so probably a NULL here means the node has been dropped from the gossip_nodes internal table entirely.
• A question from @davepacheco was whether there's anything else about these active-nodes-with-some-NULLS that give us confidence it's okay to decommission them.

The details of the reproduction make the body too long, so I'll add them in a comment. There are a few things to fix:

• Improve the warnings reported by the background execution task; show the actual errors!
• Make the CSV parsing more flexible (possibilities: make most/all of the fields Option, change the field types we only report and don't need to interpret to just String, ...)
• Update the safety checks to cope with fields it cares about being absent; it appears based on the above it's okay for us to treat a gossiped_replicas value of NULL as 0 for the purposes of that safety. It doesn't look like ranges_underreplicated becomes NULL, so maybe it's okay to keep that field as required in the parsing?

An alternative to consider is pulling this data from the Prometheus metrics coming in #8379 instead of shelling out to cockroach and parsing CSV; I assume that would have similar issues with fields being nullable, so most of the work here is probably orthogonal to the data source.

Start with nexus/reconfigurator/execution/src/omicron_zones.rs to inspect how decommission failures are summarized, then read cockroach-admin/types/src/lib.rs and the cockroach node status --all --format csv parsing path. Review the reproduction output for nullable fields on active nodes. Done means the task reports useful underlying errors, parsing accepts the observed output, and safety checks handle absent fields as described.