cockroachdb / cockroachdb/cockroach
ci: IBM s390x unit test agent is resource-constrained, causing frequent test timeouts
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Summary
The IBM s390x TeamCity build agent is significantly resource-constrained compared to GCE, causing frequent unit test timeout failures. These failures happen before the go test binary executes creating another tangential issue with test reporting and github issue routing described here https://github.com/cockroachdb/cockroach/issues/159708
Examples:
- #165212 — `pkg/sql/opt/opbench`
- #165213 — `pkg/sql/opt/optgen/exprgen`
- #165487 — `pkg/sql/gcjob`
- #165488 — `pkg/ccl/changefeedccl/kvfeed`
## Root Cause
Comparing the [IBM](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsIbmCloudLinuxS390x_UnitTestsS390x/21215148?buildTab=perfmon)'s performance monitor to [GCE](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_UnitTests_BazelUnitTests/21240612?buildTab=perfmon)'s performance monitor:
- **CPU**: IBM has **16 cores** vs GCE's **48 cores**. CPU utilization and CPU time on disk I/O are much higher on IBM.
- **Memory**: IBM build agent memory usage was observed at **~90%** during builds.
- **Scaling limitation**: The 16-core VM is the **largest available** for s390x on IBM Cloud — we cannot simply increase the instance size.
## Impact
- Frequent false-positive test failures on master that must be manually triaged and routed.
- All such failures get assigned to test-eng as a catch-all since the ambiguous `test.xml` prevents proper team routing (#159708).
- Noise in CI makes it harder to identify real regressions.
## Possible Mitigations
Note: These were generated by Claude, and I added my comments.
Currently we cannot increase the IBM VM's number of cores or RAM , potential alternatives include:
- **Reduce parallelism**: Lower the number of concurrent test targets on the s390x build to reduce CPU/memory pressure.
- Although Ricky noted "but the test suite already takes 4 hours to run. We'll be extending it much longer in that case."
- **Increase bazel test timeouts**: Give s390x tests more time to account for the slower hardware.
- Not sure there's a sustainable way to enforce this on a per CPU architecture level though
- **Shard the test suite**: Split the s390x unit test run across multiple VMs/builds.
- Currently only have a single IBM TC Agent. If we were had more agents, and we are able to tolerate longer build execution times, this may be the best way forward
- **Skip known resource-intensive tests**: Exclude tests that consistently timeout on s390x if they are not architecture-sensitive.
- We shouldn't be skipping tests
- **Optimize test resource usage**: Identify the heaviest tests and see if they can be made lighter.
- This is not an issue of a subset of heavy tests
Thread: https://cockroachlabs.slack.com/archives/C03SG8QKYRJ/p1773350909837859
/cc @cockroachdb/test-eng
Jira issue: CRDB-61453
Contributor guide
Assessment
This issue has not been assessed yet.