jenkinsci / jenkinsci/matrix-project-plugin
[JENKINS-8724] Matrix Job Parents and Executors
- Dominant language
- Java
- Stars
- 17
- Forks
- 85
- Avg merge
- 4h 27m
- Merged PRs (30d)
- 5
Description
Since JENKINS-936 matrix parent job will no longer consume an executor. While this prevents deadlocks, a problem arises when many matrix jobs are started in parallel. All parents will run e.g. doing the SVN checkouts. regular jobs consuming an executor may still run in parallel. So the overalll number of parallel running jobs/activities can easily exceed the configured number of executors - and the result is a huge overall slowdown of the system.
As a further improvement I suggest changing the behavior again. A matrix patent should start consuming an executor. It will stay in the queue if there is no free executor. Once the matrix parent is done (e.g. checking out files) and activated child jobs, it will release its executor. Thus the next build in the queue can run, which may be the child job or whatever is there. This way there will be no deadlocks and at no time more active jobs are running than executors are available.
---
Originally reported by axelheider, imported from: Matrix Job Parents and Executors
Raw content of original issue
Since
JENKINS-936matrix parent job will no longer consume an executor. While this prevents deadlocks, a problem arises when many matrix jobs are started in parallel. All parents will run e.g. doing the SVN checkouts. regular jobs consuming an executor may still run in parallel. So the overalll number of parallel running jobs/activities can easily exceed the configured number of executors - and the result is a huge overall slowdown of the system.
As a further improvement I suggest changing the behavior again. A matrix patent should start consuming an executor. It will stay in the queue if there is no free executor. Once the matrix parent is done (e.g. checking out files) and activated child jobs, it will release its executor. Thus the next build in the queue can run, which may be the child job or whatever is there. This way there will be no deadlocks and at no time more active jobs are running than executors are available.
- environment:
Windows, 1.395
Contributor guide
Research direction
No files, tests, or entry points are named. Start by tracing how matrix parent jobs enter the executor queue, perform checkout work, activate child jobs, and release executors; verify the result with parallel matrix builds and configured executor limits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100