roachtest: replicate/wide failed

roachtest.replicate/wide [failed](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAzureBazel/21518916?buildTab=log) with [artifacts](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestNightlyAzureBazel/21518916?buildTab=artifacts#/replicate/wide) on master @ [17d4d0939905dacd9f19fb6a372bed8ac62a73fa](https://github.com/cockroachdb/cockroach/commits/17d4d0939905dacd9f19fb6a372bed8ac62a73fa):

Failed with:

```
(test_runner.go:2023).runTest: test timed out (30m0s)
test artifacts and logs in: /artifacts/replicate/wide/run_1
```
Fatal entries found in Cockroach logs:

```
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 disk stall detected: disk slowness detected: syncdata on file /mnt/data1/cockroach/000079.log has been ongoing for 23.6s
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹Time Device Name Reads Completed Reads Merged Sectors Read Read Duration Writes Completed Writes Merged Sectors Written Write Duration IO in Progress IO Duration Weighted IO Duration Discards Completed Discards Merged Sectors Discarded Discard Duration Flushes Completed Flush Duration Error›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:25.431051138Z 7 sd 0 0 0 0s 550 280 6664 29ms 1 31ms 29ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:25.5310005Z 7 sd 0 0 0 0s 510 257 6152 29ms 0 27ms 29ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:25.631026858Z 7 sd 0 0 0 0s 415 230 5176 24ms 0 23ms 24ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:25.731002992Z 7 sd 0 0 0 0s 446 174 4960 25ms 0 23ms 25ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:25.830992858Z 7 sd 0 0 0 0s 476 234 6752 33ms 0 27ms 33ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:25.930984956Z 7 sd 0 0 0 0s 384 150 4272 22ms 0 23ms 22ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:26.030988268Z 7 sd 0 0 0 0s 440 172 4896 25ms 0 23ms 25ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:26.131040674Z 7 sd 0 0 0 0s 391 150 4328 23ms 1 26ms 23ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 disk stall detected: disk slowness detected: syncdata on file /mnt/data1/cockroach/000079.log has been ongoing for 23.6s
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹Time Device Name Reads Completed Reads Merged Sectors Read Read Duration Writes Completed Writes Merged Sectors Written Write Duration IO in Progress IO Duration Weighted IO Duration Discards Completed Discards Merged Sectors Discarded Discard Duration Flushes Completed Flush Duration Error›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:25.431051138Z 7 sd 0 0 0 0s 550 280 6664 29ms 1 31ms 29ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:25.5310005Z 7 sd 0 0 0 0s 510 257 6152 29ms 0 27ms 29ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:25.631026858Z 7 sd 0 0 0 0s 415 230 5176 24ms 0 23ms 24ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:25.731002992Z 7 sd 0 0 0 0s 446 174 4960 25ms 0 23ms 25ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:25.830992858Z 7 sd 0 0 0 0s 476 234 6752 33ms 0 27ms 33ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:25.930984956Z 7 sd 0 0 0 0s 384 150 4272 22ms 0 23ms 22ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:26.030988268Z 7 sd 0 0 0 0s 440 172 4896 25ms 0 23ms 25ms 0 0 0 0s 0 0s nil›
F260916 09:22:01.472178 111403 storage/pebble.go:1639 ⋮ [n3,s3,pebble] 175 +‹2026-09-16T09:21:26.131040674Z 7 sd 0 0 0 0s 391 150 4328 23ms 1 26ms 23ms 0 0 0 0s 0 0s nil›
```
Cluster Node to Ip Mapping:

| Node | Public IP | Private IP |
| --- | --- | --- |
| teamcity-21518916-1789538064-97-n9cpu1-0001 | 20.127.237.81 | 10.1.0.155 |
| teamcity-21518916-1789538064-97-n9cpu1-0002 | 20.127.144.66 | 10.1.0.102 |
| teamcity-21518916-1789538064-97-n9cpu1-0003 | 52.146.88.55 | 10.1.0.152 |
| teamcity-21518916-1789538064-97-n9cpu1-0004 | 138.91.109.181 | 10.1.0.125 |
| teamcity-21518916-1789538064-97-n9cpu1-0005 | 40.90.253.135 | 10.1.0.151 |
| teamcity-21518916-1789538064-97-n9cpu1-0006 | 20.102.99.98 | 10.1.0.153 |
| teamcity-21518916-1789538064-97-n9cpu1-0007 | 40.90.253.136 | 10.1.0.126 |
| teamcity-21518916-1789538064-97-n9cpu1-0008 | 20.127.169.95 | 10.1.0.156 |
| teamcity-21518916-1789538064-97-n9cpu1-0009 | 20.106.198.89 | 10.1.0.157 |

Parameters:
- arch=amd64
- cloud=azure
- coverageBuild=false
- cpu=1
- diskCount=0
- encrypted=false
- fs=ext4
- localSSD=true
- metamorphicLeases=default
- runtimeAssertionsBuild=false
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-21518916/replicate-wide/1789550437430/1789552399037)

See: [Datadog Logs](https://us5.datadoghq.com/logs?query=service%3Aroachtest+%40cluster%3Ateamcity-21518916-1789538064-97-n9cpu1&from_ts=1789550437430&to_ts=1789552399037&live=false&storage=flex_tier)

/cc @cockroachlabs/kv-triage

[Improve this report!](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/bazci/githubpost/issues)

Jira issue: CRDB-68450

Start with the roachtest replicate/wide failure and its artifacts in /artifacts/replicate/wide/run_1, then inspect the fatal disk-stall entries at storage/pebble.go:1639. Compare the Grafana and Datadog links with the timeout and Azure cluster parameters; done means identifying and addressing the cause of the failed test or documenting why it is environmental.

Written by the indexing model from the issue text.