cockroachdb / cockroachdb/cockroach
tests: recurring fsync/Mkdirat hangs on s390 causing failures
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
Over the past ~3 weeks (since ~2026-05-21) we've seen awave of s390x unit test timeouts whose goroutine dumps show the test goroutine blocked indefinitely in a filesystem syscall — most commonly `syscall.Fsync`, but also `syscall.Mkdirat`. The failures span unrelated tests/packages and multiple release branches (master, release-26.1, release-26.2, release-26.2.x-rc, release-25.4), which points to an infrastructure-level disk I/O stall on the single IBM Cloud s390x TeamCity agent (`teamcity-agent-2404`) rather than a code regression.
Looking at IBM Cloud metrics for the past 2 weeks does not show a clear negative trend in CPU/IO utilization [slack]: https://cockroachlabs.slack.com/archives/C023S0V4YEB/p1780652411779589.
See https://github.com/cockroachdb/cockroach/issues/170941#issuecomment-4548073833 for a representative trace.
```
goroutine 2970 [syscall]:
syscall.Mkdirat(0xffffffffffffff9c, {0xc000a802a0, 0x55}, 0x1c0)
GOROOT/src/syscall/zsyscall_linux_s390x.go:665
```
| Issue | Date | Branch | Test |
|-------|------|--------|------|
| #171644 | 2026-06-13 | release-26.2.3-rc | `storage: TestMVCCResolveTxnRangeResumeWithManyVersions` |
| #171643 | 2026-06-13 | release-26.2.3-rc | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterSize` |
| #171640 | 2026-06-13 | master | `storage: TestPebbleClusterVersionTooNew` |
| #171639 | 2026-06-13 | master | `storage: TestMVCCStatsPutWaitDeleteGC` |
| #171497 | 2026-06-09 | release-26.2 | `kv/kvserver/batcheval: TestDBAddSSTable` |
| #171447 | 2026-06-04 | release-26.1 | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterInitSST` |
| #171374 | 2026-06-03 | master | `kv/kvserver/batcheval: TestDBAddSSTable` |
| #171253 | 2026-06-03 | master | `storage: TestPebbleIterator_Corruption` |
| #171230 | 2026-05-30 | master | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterSize` |
| #171217 | 2026-05-29 | master | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterSize` |
| #171076 | 2026-05-29 | release-26.2 | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterInitSST` |
| #171045 | 2026-05-27 | release-25.4 | `kv/kvserver/batcheval: TestDBAddSSTable` |
| #170970 | 2026-05-27 | release-25.4 | `kv/kvserver/batcheval: TestDBAddSSTable` |
| #170941 | 2026-05-26 | release-26.2 | `storage: TestPebbleIterReuse` (Mkdirat) |
| #170940 | 2026-05-26 | release-26.2 | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterSize` |
| #170906 | 2026-05-25 | master | `storage: TestMVCCDeleteRangeWithSequence` |
| #170905 | 2026-05-25 | master | `storage: TestMVCCHistories` |
| #170904 | 2026-05-25 | master | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterInitSST` |
| #170888 | 2026-05-25 | master | `kv/kvserver/batcheval: TestRandomKeyAndTimestampExport` |
| #170857 | 2026-05-24 | master | `storage: TestPebbleIterator_Corruption` |
| #170846 | 2026-05-23 | release-26.2 | `ts: TestQueryNearCurrentTime` |
| #170795 | 2026-05-22 | release-26.2 | `storage: TestNewTempEngine` |
| #170770 | 2026-05-21 | release-25.4 | `storage: TestMVCCGarbageCollectClearPointsInRange` |
| #170769 | 2026-05-21 | release-25.4 | `storage: TestPebbleBackgroundError` |
| #170699 | 2026-05-21 | release-26.2.1-rc | `ts: TestServerQueryMemoryManagement` |
Jira issue: CRDB-64789
Contributor guide
Assessment
This issue has not been assessed yet.