OWASP / OWASP/SecurityShepherd
perf: parallelize integration tests via GitHub Actions matrix sharding (4 shards)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 515
- Avg merge
- 3h 46m
- Merged PRs (30d)
- 1
Description
Problem
The integration-tests job in .github/workflows/test.yml takes ~4 minutes. All 40 *IT classes run serially in a single JVM against one shared MySQL instance.
Background — what's already been done
This builds on prior perf work, do not redo it:
- #831 (DONE, merged) — DDL-once + per-class data reseed. The schema is now built once per JVM (
TestProperties.ensureSchemaReady) and each class does a cheapreseedTestData()in@BeforeAllinstead of a full ~3,280-line schema rebuild. This is the code currently in src/test/java/testUtils/TestProperties.java. It already cut runtime substantially (~5 min → ~4 min). - #828 (superseded by #831) — single DB init + targeted resets. Notably, #828 listed "Parallel execution (for now)" as an explicit Non-Goal — "introduces unnecessary complexity for current scale." This issue is the intended follow-up to pick that up now that the per-class reset machinery exists.
So the remaining lever is no longer per-class cost (already minimized) — it's serial execution. We parallelize.
Why naive thread-parallelism is unsafe
The 40 classes share global state: one MySQL core schema plus in-memory static caches (ScoreboardStatus, CheatSheetStatus, CountdownHandler, FeedbackStatus, OpenRegistration, ModulePlan). Running them concurrently in one JVM/DB would corrupt each other's state. reseedTestData() guarantees isolation only between serial classes, not concurrent ones.
Recommended approach: GitHub Actions matrix sharding (4 shards)
Split the 40 IT classes across 4 parallel jobs, each on its own runner with its own MySQL service container (4 MySQL instances total, not 40). Because each shard is a separate JVM + separate DB, the shared-state problem disappears with no test-code changes. Expected wall-clock ~4 min → ~1–1.5 min (cost: ~4× runner-minutes).
Failsafe honors -Dit.test=<comma-separated globs> on the command line, so shards can be selected by package glob without maintaining hardcoded class lists. Suggested balanced grouping:
| Shard | Globs (-Dit.test=) |
~classes |
|---|---|---|
| 1 | dbProcs.*IT, servlets.LoginIT, servlets.LogoutIT, servlets.SetupIT, testUtils.*IT |
7 (incl. heavy Getter/Setter) |
| 2 | servlets.admin.** |
18 (small/fast each) |
| 3 | servlets.module.lesson.*IT |
9 |
| 4 | servlets.api.*IT, servlets.module.GetModuleIT, servlets.module.challenge.*IT |
6 |
Implementation sketch: turn the job into a strategy.matrix over the 4 shards (with fail-fast: false), keep the existing services.mysql + setup steps per shard, and append -Dit.test=${{ matrix.shard.tests }} to the existing mvn verify step.
Alternatives considered
- Failsafe
forkCount+ per-fork schema (${surefire.forkNumber}→core_1,core_2…): keeps it to one runner (no extra cost) but requires making the DB/schema name configurable inTestProperties(currently hardcoded"core"at TestProperties.java:494) and provisioning N schemas. More invasive than matrix sharding. - Cheaper wins regardless: share the compiled artifact between the
buildand IT jobs instead of recompiling per shard; verify the Maven cache is actually hitting.
Acceptance criteria
- IT wall-clock time meaningfully reduced (target ~1–1.5 min).
- All 40 IT classes still execute — no shard gaps; verify total test count matches pre-change.
- Adding a new IT class is easy to route to a shard (prefer package-glob over hardcoded class names).
-
fail-fast: falseso one shard failing still reports the others. - No production code changes (matrix approach); if forkCount is chosen instead, only
TestPropertieschanges.
Refs: Maven Failsafe — fork options & parallel execution, GitHub Actions matrix strategy
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the integration-tests job in .github/workflows/test.yml around lines 91-123, then inspect the existing TestProperties setup only to understand the isolation already provided by separate JVMs and MySQL services. Run the current integration-test command to establish the baseline, then verify the matrix runs all 40 IT classes across four shards, uses fail-fast: false, and reduces wall-clock time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, java, mysql
- Domain
- ci-cd, devops, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100