roachtest: db-console/mixed-version-endpoints failed

roachtest.db-console/mixed-version-endpoints [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAzureBazel/21513552?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAzureBazel/21513552?buildTab=artifacts#/db-console/mixed-version-endpoints) on master @ [2bf53f5e2960d67b69a8d3576c03e3e9fc3dae2a](https://github.com/cockroachdb/cockroach/commits/2bf53f5e2960d67b69a8d3576c03e3e9fc3dae2a):

Failed with:

```
(test_monitor.go:35).func1: test monitor: unexpected node event: n4: cockroach process for system interface died (exit code 7)
(mixedversion.go:1037).Run: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/db-console/mixed-version-endpoints/run_1
```
Fatal entries found in Cockroach logs:

```
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 disk stall detected: unable to sync log files within 20s
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 !goroutine 31732 [running]:
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0xd6ac39?, 0x8919420?})
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 ! pkg/util/allstacks/allstacks.go:19
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 ! pkg/util/log/clog.go:294 +0xc6
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x8919420, 0xf7fce80}, 0x2, 0x4, 0x1, 0x1?, {0x84f364b, 0x37}, {0x3a01aad6dfc0, 0x1, ...})
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 ! pkg/util/log/channels.go:104 +0x650
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 disk stall detected: unable to sync log files within 20s
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 !goroutine 31732 [running]:
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 !github.com/cockroachdb/cockroach/pkg/util/allstacks.GetWithBuf({0x0?, 0xd6ac39?, 0x8919420?})
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 ! pkg/util/allstacks/allstacks.go:38 +0x125
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 !github.com/cockroachdb/cockroach/pkg/util/allstacks.Get(...)
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 ! pkg/util/allstacks/allstacks.go:19
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 ! pkg/util/log/clog.go:294 +0xc6
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x8919420, 0xf7fce80}, 0x2, 0x4, 0x1, 0x1?, {0x84f364b, 0x37}, {0x3a01aad6dfc0, 0x1, ...})
F260912 07:10:30.859174 31732 1@util/log/file.go:270 ⋮ [-] 967 ! pkg/util/log/channels.go:104 +0x650
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 disk stall detected: disk slowness detected: syncdata on file /mnt/data1/cockroach/000704.log has been ongoing for 22.0s
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹Time Device Name Reads Completed Reads Merged Sectors Read Read Duration Writes Completed Writes Merged Sectors Written Write Duration IO in Progress IO Duration Weighted IO Duration Discards Completed Discards Merged Sectors Discarded Discard Duration Flushes Completed Flush Duration Error›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.1240672Z 1 0 0 0 0s 15 21 1624 7ms 0 5ms 7ms 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.224146493Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.324159383Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.424513587Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.523643309Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.623844716Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.723865783Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.824089102Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 disk stall detected: disk slowness detected: syncdata on file /mnt/data1/cockroach/000704.log has been ongoing for 22.0s
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹Time Device Name Reads Completed Reads Merged Sectors Read Read Duration Writes Completed Writes Merged Sectors Written Write Duration IO in Progress IO Duration Weighted IO Duration Discards Completed Discards Merged Sectors Discarded Discard Duration Flushes Completed Flush Duration Error›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.1240672Z 1 0 0 0 0s 15 21 1624 7ms 0 5ms 7ms 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.224146493Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.324159383Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.424513587Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.523643309Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.623844716Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.723865783Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
F260912 07:10:22.963233 5070 storage/pebble.go:1489 ⋮ [n4,s4,pebble] 251 +‹2026-09-12T07:09:53.824089102Z 1 0 0 0 0s 0 0 0 0s 0 0s 0s 0 0 0 0s 0 0s nil›
```
Cluster Node to Ip Mapping:

| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21513552-1789192537-55-n5cpu4-0001 | 172.203.250.162 | 10.1.0.137 |
| teamcity-21513552-1789192537-55-n5cpu4-0002 | 20.102.44.90 | 10.1.0.135 |
| teamcity-21513552-1789192537-55-n5cpu4-0003 | 20.127.232.116 | 10.1.0.127 |
| teamcity-21513552-1789192537-55-n5cpu4-0004 | 57.151.99.164 | 10.1.0.136 |
| teamcity-21513552-1789192537-55-n5cpu4-0005 | 74.235.188.199 | 10.1.0.155 |

Parameters:
- arch=amd64
- cloud=azure
- coverageBuild=false
- cpu=4
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- mvtDeploymentMode=shared-process
- mvtVersions=v26.3.1 → v26.4.0-alpha.2-dev-2bf53f5e2960d67b69a8d3576c03e3e9fc3dae2a (master)
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21513552/db-console-mixed-version-endpoints/1789196793033/1789197785232)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21513552-1789192537-55-n5cpu4&from_ts=1789196793033&to_ts=1789197785232&live=false&storage=flex_tier)

/cc @cockroachlabs/obs-prs

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68285

Start with the db-console/mixed-version-endpoints artifacts and logs, then inspect the failure paths in test_monitor.go:35, mixedversion.go:1037, and storage/pebble.go:1489. Reproduce the Azure roachtest failure and determine why node n4 exits during the reported disk stall; done means the test completes without the system interface process dying.

Written by the indexing model from the issue text.