dotCMS / dotCMS/core

CI: rebalance integration and Postman shards (deferred — needs test-independence work first)

Open
#36,964 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Team : Maintenance Type : Task
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Split out of #36943 so the independent fixes there could land. The work is preserved on branch issue-36942-shard-rebalance-wip at f612b27ed8 — resume from there rather than rebuilding it.

It works

Measured on run 31203516115, with fail-fast disabled so every shard reported instead of the first failure cancelling the matrix:

IT shards    22.2 - 26.2m   (was 38.9m)
Postman      18.6 - 28.8m   (was 37.8m)

Bin-packing on measured per-class time balanced 557 integration classes to 18.4m each, spread 0.0m. Per-class times are stable run-to-run (115.2m vs 114.8m on a control set), so the balance should hold. Job wall times are noisy — an untouched shard swung +8m between runs — partly because runners vary by VM generation (D4ads_v5 / D4ds_v6 / D4ds_v7, all 4 vCPU / 15 GB).

Why it is not merged

3 of 7 integration shards and 5 of 9 Postman shards failed. Every failure was a test that only passed because of which suite-mates ran before it.

The Postman groups are dependency clusters, not labels. category-content grouped Category + ContentResourceV1 + Content_Resource because those collections share data. Rebalancing purely on time shattered every group. The GraphQL folder split failed the same way: "Page API - Testing 'page' field with inline fragments" needs setup performed by the Page API folder, which landed in the other shard.

Why it is still worth doing

Beyond the ~12m of wall clock, it is an effective detector of tests that depend on their neighbours. Every failure it surfaced was a real defect, not a false positive:

  • AWSS3PublishingEndPoint threw a NullPointerException from inside a catch block whenever no request was bound to the thread — fixed in #36943
  • The Language Variable content type is deletable when it should be system — #36958
  • RuleBundlerTest created fixtures in a static @DataProvider, evaluated at suite construction, ~65 classes before use — fixed in #36943
  • PublishingEndPointTest depended on a leaked HttpServletRequestThreadLocal — fixed in #36943

Outstanding, found but not yet fixed:

  • FieldUtilTest, ContentletAjaxTest — assume a suite-mate materialised the Languagevariable content type
  • MenuResourceTest.test_get_translation — language-variable lookup
  • WebAssetHelperIntegrationTest ×3 — limited-user permissions
  • AjaxDirectorServletIntegrationTest
  • PersonaAPITest — asserts an absolute global count (allPersonas should be 5, got:6), so any test that leaves a persona behind breaks it

Suggested approach when resumed

  1. Fix the tests above on their own, against the current shard layout, before re-sharding anything. Each is a real bug and lands independently.
  2. For Postman, treat existing groups as atomic. Rebalance by moving whole groups between shards, or by splitting a single oversized collection by file (as default-split was done historically) — not by regrouping collections across clusters.
  3. For integration, consider contiguous splits of the largest suites, which preserve each class's neighbours, before free-form repacking.
  4. Keep fail-fast: false while iterating — otherwise each ~40 minute run reveals only one shard's worth of coupling. Revert it before merging.

Note on measurement

.github/scripts/test-balance/find_swallowed.py landed in #36943. The packing scripts (parse_suites.py, pack_suites.py, pack_postman.py) are on the preserved branch.

The rebalance only pays off if both tails move: integration and Postman maxima were within a minute of each other (38.9m vs 37.8m), so fixing one alone changes wall clock by ~0.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Resume from branch issue-36942-shard-rebalance-wip at f612b27ed8. Read .github/scripts/test-balance/find_swallowed.py, parse_suites.py, pack_suites.py, and pack_postman.py, then run the current shard layout with fail-fast disabled to investigate the listed test failures. Done means test-independence fixes land, Postman groups remain atomic, integration neighbours stay together, both shard tails improve, and fail-fast is restored before merging.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postman, python
Domain
ci-cd, testing-qa
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.