cockroachdb / cockroachdb/cockroach

ci: IBM s390x unit test agent is resource-constrained, causing frequent test timeouts

Open
#165,583 0 comments 0 reactions 0 assignees View on GitHub
A-testeng-infra C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Summary

The IBM s390x TeamCity build agent is significantly resource-constrained compared to GCE, causing frequent unit test timeout failures. These failures happen before the go test binary executes creating another tangential issue with test reporting and github issue routing described here https://github.com/cockroachdb/cockroach/issues/159708

Examples:
- #165212 — `pkg/sql/opt/opbench`
- #165213 — `pkg/sql/opt/optgen/exprgen`
- #165487 — `pkg/sql/gcjob`
- #165488 — `pkg/ccl/changefeedccl/kvfeed`

## Root Cause

Comparing the [IBM](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_TestsIbmCloudLinuxS390x_UnitTestsS390x/21215148?buildTab=perfmon)'s performance monitor to [GCE](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_UnitTests_BazelUnitTests/21240612?buildTab=perfmon)'s performance monitor:

- **CPU**: IBM has **16 cores** vs GCE's **48 cores**. CPU utilization and CPU time on disk I/O are much higher on IBM.
- **Memory**: IBM build agent memory usage was observed at **~90%** during builds.
- **Scaling limitation**: The 16-core VM is the **largest available** for s390x on IBM Cloud — we cannot simply increase the instance size.

## Impact

- Frequent false-positive test failures on master that must be manually triaged and routed.
- All such failures get assigned to test-eng as a catch-all since the ambiguous `test.xml` prevents proper team routing (#159708).
- Noise in CI makes it harder to identify real regressions.

## Possible Mitigations
Note: These were generated by Claude, and I added my comments.
Currently we cannot increase the IBM VM's number of cores or RAM , potential alternatives include:
- **Reduce parallelism**: Lower the number of concurrent test targets on the s390x build to reduce CPU/memory pressure.
- Although Ricky noted "but the test suite already takes 4 hours to run. We'll be extending it much longer in that case."
- **Increase bazel test timeouts**: Give s390x tests more time to account for the slower hardware.
- Not sure there's a sustainable way to enforce this on a per CPU architecture level though
- **Shard the test suite**: Split the s390x unit test run across multiple VMs/builds.
- Currently only have a single IBM TC Agent. If we were had more agents, and we are able to tolerate longer build execution times, this may be the best way forward
- **Skip known resource-intensive tests**: Exclude tests that consistently timeout on s390x if they are not architecture-sensitive.
- We shouldn't be skipping tests
- **Optimize test resource usage**: Identify the heaviest tests and see if they can be made lighter.
- This is not an issue of a subset of heavy tests

Thread: https://cockroachlabs.slack.com/archives/C03SG8QKYRJ/p1773350909837859

/cc @cockroachdb/test-eng

Jira issue: CRDB-61453

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.