OWASP / OWASP/SecurityShepherd

perf: parallelize integration tests via GitHub Actions matrix sharding (4 shards)

Open
#863 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Enhancement github_actions Test Automation
Dominant language
Java
Stars
1.5k
Forks
515
Avg merge
3h 46m
Merged PRs (30d)
1

Description

Problem

The integration-tests job in .github/workflows/test.yml takes ~4 minutes. All 40 *IT classes run serially in a single JVM against one shared MySQL instance.

Background — what's already been done

This builds on prior perf work, do not redo it:

  • #831 (DONE, merged) — DDL-once + per-class data reseed. The schema is now built once per JVM (TestProperties.ensureSchemaReady) and each class does a cheap reseedTestData() in @BeforeAll instead of a full ~3,280-line schema rebuild. This is the code currently in src/test/java/testUtils/TestProperties.java. It already cut runtime substantially (~5 min → ~4 min).
  • #828 (superseded by #831) — single DB init + targeted resets. Notably, #828 listed "Parallel execution (for now)" as an explicit Non-Goal — "introduces unnecessary complexity for current scale." This issue is the intended follow-up to pick that up now that the per-class reset machinery exists.

So the remaining lever is no longer per-class cost (already minimized) — it's serial execution. We parallelize.

Why naive thread-parallelism is unsafe

The 40 classes share global state: one MySQL core schema plus in-memory static caches (ScoreboardStatus, CheatSheetStatus, CountdownHandler, FeedbackStatus, OpenRegistration, ModulePlan). Running them concurrently in one JVM/DB would corrupt each other's state. reseedTestData() guarantees isolation only between serial classes, not concurrent ones.

Recommended approach: GitHub Actions matrix sharding (4 shards)

Split the 40 IT classes across 4 parallel jobs, each on its own runner with its own MySQL service container (4 MySQL instances total, not 40). Because each shard is a separate JVM + separate DB, the shared-state problem disappears with no test-code changes. Expected wall-clock ~4 min → ~1–1.5 min (cost: ~4× runner-minutes).

Failsafe honors -Dit.test=<comma-separated globs> on the command line, so shards can be selected by package glob without maintaining hardcoded class lists. Suggested balanced grouping:

Shard Globs (-Dit.test=) ~classes
1 dbProcs.*IT, servlets.LoginIT, servlets.LogoutIT, servlets.SetupIT, testUtils.*IT 7 (incl. heavy Getter/Setter)
2 servlets.admin.** 18 (small/fast each)
3 servlets.module.lesson.*IT 9
4 servlets.api.*IT, servlets.module.GetModuleIT, servlets.module.challenge.*IT 6

Implementation sketch: turn the job into a strategy.matrix over the 4 shards (with fail-fast: false), keep the existing services.mysql + setup steps per shard, and append -Dit.test=${{ matrix.shard.tests }} to the existing mvn verify step.

Alternatives considered

  • Failsafe forkCount + per-fork schema (${surefire.forkNumber}core_1, core_2…): keeps it to one runner (no extra cost) but requires making the DB/schema name configurable in TestProperties (currently hardcoded "core" at TestProperties.java:494) and provisioning N schemas. More invasive than matrix sharding.
  • Cheaper wins regardless: share the compiled artifact between the build and IT jobs instead of recompiling per shard; verify the Maven cache is actually hitting.

Acceptance criteria

  • IT wall-clock time meaningfully reduced (target ~1–1.5 min).
  • All 40 IT classes still execute — no shard gaps; verify total test count matches pre-change.
  • Adding a new IT class is easy to route to a shard (prefer package-glob over hardcoded class names).
  • fail-fast: false so one shard failing still reports the others.
  • No production code changes (matrix approach); if forkCount is chosen instead, only TestProperties changes.

Refs: Maven Failsafe — fork options & parallel execution, GitHub Actions matrix strategy

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the integration-tests job in .github/workflows/test.yml around lines 91-123, then inspect the existing TestProperties setup only to understand the isolation already provided by separate JVMs and MySQL services. Run the current integration-test command to establish the baseline, then verify the matrix runs all 40 IT classes across four shards, uses fail-fast: false, and reduces wall-clock time.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, java, mysql
Domain
ci-cd, devops, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.