Intermittent ConcurrentModificationException in parallel jobDsl job-generation step
- Dominant language
- HTML
- Stars
- 155
- Forks
- 346
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 36
Description
Summary:
---
When a test target splits into many parallel testLists (e.g. JCK extended runs with 25-27 branches), each branch independently calls jobDsl to generate its child job. These calls run fully concurrently, and intermittently race on writing to the shared `GeneratedObjectsRunAction#modifiedObjects` map while Jenkins is mid-checkpoint, causing the build to fail with FAILURE even though job generation was otherwise working correctly.
Evidence:
---
Two independent failures with the identical stack trace, on different platforms/JDK versions:
[Test_openjdk21_j9_extended.jck_aarch64_linux #169 ](https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk21_j9_extended.jck_aarch64_linux/169/console)(25 branches)
Test_openjdk25_j9_extended.jck_aarch64_linux #[63](https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk25_j9_extended.jck_aarch64_linux/63/console) (27 branches, failed specifically in testList_13)
Root cause:
---
JenkinsfileBase calls jobDsl targets: templatePath, ... once per parallel branch with no synchronization between them. The Job DSL plugin's GeneratedObjectsRunAction isn't thread-safe across concurrent jobDsl invocations within the same build, which is a known issue in other job-dsl-in-parallel setups.
Impact:
---
Nondeterministic — failure depends on timing, so the same job can pass on one run and fail on the next with no code change, making it confusing to triage and wasting machine time on rebuilds.
Suggested fix:
---
Wrap the jobDsl call in JenkinsfileBase in a lock() block scoped per-build (e.g. lock("jobdsl-${env.BUILD_TAG}")) so branches within the same build serialize against each other without contending with unrelated concurrent builds across the Jenkins instance. But test it , to check the time it takes is it slower when serialized?
Contributor guide
Research direction
Start in JenkinsfileBase at the jobDsl invocation made by each parallel testList branch. Reproduce a run with many branches and inspect the console output for ConcurrentModificationException during job generation. Serialize the per-build jobDsl calls as proposed, then compare repeated-run stability and build duration before and after the change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy
- Domain
- build-system, ci-cd, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100