CI: rebalance integration and Postman shards (deferred — needs test-independence work first)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Split out of #36943 so the independent fixes there could land. The work is preserved on branch issue-36942-shard-rebalance-wip at f612b27ed8 — resume from there rather than rebuilding it.
It works
Measured on run 31203516115, with fail-fast disabled so every shard reported instead of the first failure cancelling the matrix:
IT shards 22.2 - 26.2m (was 38.9m)
Postman 18.6 - 28.8m (was 37.8m)
Bin-packing on measured per-class time balanced 557 integration classes to 18.4m each, spread 0.0m. Per-class times are stable run-to-run (115.2m vs 114.8m on a control set), so the balance should hold. Job wall times are noisy — an untouched shard swung +8m between runs — partly because runners vary by VM generation (D4ads_v5 / D4ds_v6 / D4ds_v7, all 4 vCPU / 15 GB).
Why it is not merged
3 of 7 integration shards and 5 of 9 Postman shards failed. Every failure was a test that only passed because of which suite-mates ran before it.
The Postman groups are dependency clusters, not labels. category-content grouped Category + ContentResourceV1 + Content_Resource because those collections share data. Rebalancing purely on time shattered every group. The GraphQL folder split failed the same way: "Page API - Testing 'page' field with inline fragments" needs setup performed by the Page API folder, which landed in the other shard.
Why it is still worth doing
Beyond the ~12m of wall clock, it is an effective detector of tests that depend on their neighbours. Every failure it surfaced was a real defect, not a false positive:
AWSS3PublishingEndPointthrew aNullPointerExceptionfrom inside a catch block whenever no request was bound to the thread — fixed in #36943- The Language Variable content type is deletable when it should be
system— #36958 RuleBundlerTestcreated fixtures in a static@DataProvider, evaluated at suite construction, ~65 classes before use — fixed in #36943PublishingEndPointTestdepended on a leakedHttpServletRequestThreadLocal— fixed in #36943
Outstanding, found but not yet fixed:
FieldUtilTest,ContentletAjaxTest— assume a suite-mate materialised theLanguagevariablecontent typeMenuResourceTest.test_get_translation— language-variable lookupWebAssetHelperIntegrationTest×3 — limited-user permissionsAjaxDirectorServletIntegrationTestPersonaAPITest— asserts an absolute global count (allPersonas should be 5, got:6), so any test that leaves a persona behind breaks it
Suggested approach when resumed
- Fix the tests above on their own, against the current shard layout, before re-sharding anything. Each is a real bug and lands independently.
- For Postman, treat existing groups as atomic. Rebalance by moving whole groups between shards, or by splitting a single oversized collection by file (as
default-splitwas done historically) — not by regrouping collections across clusters. - For integration, consider contiguous splits of the largest suites, which preserve each class's neighbours, before free-form repacking.
- Keep
fail-fast: falsewhile iterating — otherwise each ~40 minute run reveals only one shard's worth of coupling. Revert it before merging.
Note on measurement
.github/scripts/test-balance/find_swallowed.py landed in #36943. The packing scripts (parse_suites.py, pack_suites.py, pack_postman.py) are on the preserved branch.
The rebalance only pays off if both tails move: integration and Postman maxima were within a minute of each other (38.9m vs 37.8m), so fixing one alone changes wall clock by ~0.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Resume from branch issue-36942-shard-rebalance-wip at f612b27ed8. Read .github/scripts/test-balance/find_swallowed.py, parse_suites.py, pack_suites.py, and pack_postman.py, then run the current shard layout with fail-fast disabled to investigate the listed test failures. Done means test-independence fixes land, Postman groups remain atomic, integration neighbours stay together, both shard tails improve, and fail-fast is restored before merging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, postman, python
- Domain
- ci-cd, testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100