adoptium / adoptium/aqa-tests

Intermittent ConcurrentModificationException in parallel jobDsl job-generation step

Open
#7,212 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
155
Forks
346
Avg merge
2d 9h
Merged PRs (30d)
36

Description

Summary:
---

When a test target splits into many parallel testLists (e.g. JCK extended runs with 25-27 branches), each branch independently calls jobDsl to generate its child job. These calls run fully concurrently, and intermittently race on writing to the shared `GeneratedObjectsRunAction#modifiedObjects` map while Jenkins is mid-checkpoint, causing the build to fail with FAILURE even though job generation was otherwise working correctly.

Evidence:
---

Two independent failures with the identical stack trace, on different platforms/JDK versions:

[Test_openjdk21_j9_extended.jck_aarch64_linux #169 ](https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk21_j9_extended.jck_aarch64_linux/169/console)(25 branches)

Test_openjdk25_j9_extended.jck_aarch64_linux #[63](https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk25_j9_extended.jck_aarch64_linux/63/console) (27 branches, failed specifically in testList_13)

Root cause:
---

JenkinsfileBase calls jobDsl targets: templatePath, ... once per parallel branch with no synchronization between them. The Job DSL plugin's GeneratedObjectsRunAction isn't thread-safe across concurrent jobDsl invocations within the same build, which is a known issue in other job-dsl-in-parallel setups.

Impact:
---
Nondeterministic — failure depends on timing, so the same job can pass on one run and fail on the next with no code change, making it confusing to triage and wasting machine time on rebuilds.

Suggested fix:
---
Wrap the jobDsl call in JenkinsfileBase in a lock() block scoped per-build (e.g. lock("jobdsl-${env.BUILD_TAG}")) so branches within the same build serialize against each other without contending with unrelated concurrent builds across the Jenkins instance. But test it , to check the time it takes is it slower when serialized?

Contributor guide

Open the contributing guide

Research direction

Start in JenkinsfileBase at the jobDsl invocation made by each parallel testList branch. Reproduce a run with many branches and inspect the console output for ConcurrentModificationException during job generation. Serialize the per-build jobDsl calls as proposed, then compare repeated-run stability and build duration before and after the change.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
build-system, ci-cd, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.