cockroachdb / cockroachdb/cockroach

tests: recurring fsync/Mkdirat hangs on s390 causing failures

Open
#171,669 7 comments 0 reactions 1 assignee Claimed by @srosenberg View on GitHub
A-s390x A-testing C-bug P-2 T-dev-inf
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

Over the past ~3 weeks (since ~2026-05-21) we've seen awave of s390x unit test timeouts whose goroutine dumps show the test goroutine blocked indefinitely in a filesystem syscall — most commonly `syscall.Fsync`, but also `syscall.Mkdirat`. The failures span unrelated tests/packages and multiple release branches (master, release-26.1, release-26.2, release-26.2.x-rc, release-25.4), which points to an infrastructure-level disk I/O stall on the single IBM Cloud s390x TeamCity agent (`teamcity-agent-2404`) rather than a code regression.

Looking at IBM Cloud metrics for the past 2 weeks does not show a clear negative trend in CPU/IO utilization [slack]: https://cockroachlabs.slack.com/archives/C023S0V4YEB/p1780652411779589.

See https://github.com/cockroachdb/cockroach/issues/170941#issuecomment-4548073833 for a representative trace.

```
goroutine 2970 [syscall]:
syscall.Mkdirat(0xffffffffffffff9c, {0xc000a802a0, 0x55}, 0x1c0)
GOROOT/src/syscall/zsyscall_linux_s390x.go:665
```

| Issue | Date | Branch | Test |
|-------|------|--------|------|
| #171644 | 2026-06-13 | release-26.2.3-rc | `storage: TestMVCCResolveTxnRangeResumeWithManyVersions` |
| #171643 | 2026-06-13 | release-26.2.3-rc | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterSize` |
| #171640 | 2026-06-13 | master | `storage: TestPebbleClusterVersionTooNew` |
| #171639 | 2026-06-13 | master | `storage: TestMVCCStatsPutWaitDeleteGC` |
| #171497 | 2026-06-09 | release-26.2 | `kv/kvserver/batcheval: TestDBAddSSTable` |
| #171447 | 2026-06-04 | release-26.1 | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterInitSST` |
| #171374 | 2026-06-03 | master | `kv/kvserver/batcheval: TestDBAddSSTable` |
| #171253 | 2026-06-03 | master | `storage: TestPebbleIterator_Corruption` |
| #171230 | 2026-05-30 | master | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterSize` |
| #171217 | 2026-05-29 | master | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterSize` |
| #171076 | 2026-05-29 | release-26.2 | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterInitSST` |
| #171045 | 2026-05-27 | release-25.4 | `kv/kvserver/batcheval: TestDBAddSSTable` |
| #170970 | 2026-05-27 | release-25.4 | `kv/kvserver/batcheval: TestDBAddSSTable` |
| #170941 | 2026-05-26 | release-26.2 | `storage: TestPebbleIterReuse` (Mkdirat) |
| #170940 | 2026-05-26 | release-26.2 | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterSize` |
| #170906 | 2026-05-25 | master | `storage: TestMVCCDeleteRangeWithSequence` |
| #170905 | 2026-05-25 | master | `storage: TestMVCCHistories` |
| #170904 | 2026-05-25 | master | `kv/kvserver/kvstorage/snaprecv: TestMultiSSTWriterInitSST` |
| #170888 | 2026-05-25 | master | `kv/kvserver/batcheval: TestRandomKeyAndTimestampExport` |
| #170857 | 2026-05-24 | master | `storage: TestPebbleIterator_Corruption` |
| #170846 | 2026-05-23 | release-26.2 | `ts: TestQueryNearCurrentTime` |
| #170795 | 2026-05-22 | release-26.2 | `storage: TestNewTempEngine` |
| #170770 | 2026-05-21 | release-25.4 | `storage: TestMVCCGarbageCollectClearPointsInRange` |
| #170769 | 2026-05-21 | release-25.4 | `storage: TestPebbleBackgroundError` |
| #170699 | 2026-05-21 | release-26.2.1-rc | `ts: TestServerQueryMemoryManagement` |

Jira issue: CRDB-64789

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.