jenkinsci / jenkinsci/durable-task-plugin
[JENKINS-75670] cleanUpGlobalClassValue hangs entire Jenkins Server
- Dominant language
- Java
- Stars
- 30
- Forks
- 101
- PR merge metrics
- No merged PRs in 30d
Description
Hi, we are Experiencing an total Hang and Bricking of all our CI-Jobs when to many Jobs what to Finish.
According to the Logs and the built-in OS-Processes out Master tries to Cleanup GlobalClassValues add the End of Pipelines.
The Problem is takes forever to finish the Job, leading to excessive Memory and CPU usage when Java tries to run Cleanup GlobalClassValues, leading to this task blocking all other tasks including getting respones from Build Agents and Proceeding or Finishing with other Pipelines.
This prevents unutilized Resources to be freed and leads to a deadlock.
---
Originally reported by daniel_kaindlrigler, imported from: cleanUpGlobalClassValue hangs entire Jenkins Server
Raw content of original issue
Hi, we are Experiencing an total Hang and Bricking of all our CI-Jobs when to many Jobs what to Finish.
According to the Logs and the built-in OS-Processes out Master tries to Cleanup GlobalClassValues add the End of Pipelines.
The Problem is takes forever to finish the Job, leading to excessive Memory and CPU usage when Java tries to run Cleanup GlobalClassValues, leading to this task blocking all other tasks including getting respones from Build Agents and Proceeding or Finishing with other Pipelines.
This prevents unutilized Resources to be freed and leads to a deadlock.
environment
```
Jenkins: 2.504.1
OS: Linux - 5.14.0-503.40.1.el9_5.x86_64
Java: 21.0.7 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
active-directory:2.40
ansicolor:1.0.6
ant:513.vde9e7b_a_0da_0f
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
artifactory:4.0.8
artifactory-artifact-manager:222.v721418296a_b_1
artifactory-client-api:2.19.0-92.v57b_50df9c76f
asm-api:9.8-135.vb_2239d08ee90
authentication-tokens:1.131.v7199556c3004
badge:2.8
blueocean-autofavorite:1.2.5
blueocean-commons:1.27.19
blueocean-config:1.27.19
blueocean-core-js:1.27.19
blueocean-dashboard:1.27.19
blueocean-display-url:2.4.4
blueocean-events:1.27.19
blueocean-git-pipeline:1.27.19
blueocean-i18n:1.27.19
blueocean-jira:1.27.19
blueocean-jwt:1.27.19
blueocean-personalization:1.27.19
blueocean-pipeline-api-impl:1.27.19
blueocean-pipeline-editor:1.27.19
blueocean-pipeline-scm-api:1.27.19
blueocean-rest:1.27.19
blueocean-rest-impl:1.27.19
blueocean-web:1.27.19
bootstrap5-api:5.3.5-1
bouncycastle-api:2.30.1.80-256.vf98926042a_9b_
branch-api:2.1217.v43d8b_b_d8b_2c7
build-failure-analyzer:2.5.5
build-monitor-plugin:1.14-985.v7b_f37b_3d0b_f5
caffeine-api:3.2.0-166.v72a_6d74b_870f
checks-api:370.vb_61a_c57328f3
cloudbees-disk-usage-simple:241.v0ea_d9a_a_96ee8
cloudbees-folder:6.1012.v79a_86a_1ea_c1f
command-launcher:123.v37cfdc92ef67
commons-compress-api:1.27.1-3
commons-lang3-api:3.17.0-87.v5cf526e63b_8b_
commons-text-api:1.13.0-153.v91dcd89e2a_22
config-file-provider:988.v0461fcc2b_9d1
coverage:2.5.0
credentials:1415.v831096eb_5534
credentials-binding:687.v619cb_15e923f
cucumber-reports:5.8.5
dark-theme:524.vd675b_22b_30cb_
data-tables-api:2.2.2-1
dependency-track:6.0.1
display-url-api:2.209.v582ed814ff2f
docker-commons:451.vd12c371eeeb_3
docker-workflow:611.v16e84da_6d3ff
dtkit-api:3.0.3
durable-task:587.v84b_877235b_45
echarts-api:5.6.0-4
eddsa-api:0.3.0.1-19.vc432d923e5ee
email-ext:1876.v28d8d38315b_d
emailext-template:233.v1eb_88fc160b_5
emoji-symbols-api:16.0-26.v9818ff7423f0
extended-read-permission:61.vf24570ff3b_e9
external-monitor-job:223.vb_fddcf42c9b_3
favorite:2.225.v68765b_b_a_1fa_3
font-awesome-api:6.7.2-1
forensics-api:3.1.0
git:5.7.0
git-client:6.1.3
github:1.43.0
github-api:1.321-488.v9b_c0da_9533f8
github-branch-source:1815.v9152b_2ff7a_1b_
gitlab-api:5.6.0-100.v83f8f4b_f1129
gitlab-branch-source:718.v40b_5f0e67cd3
gitlab-plugin:1.9.8
google-oauth-plugin:1.335.ve6de40e2db_18
gradle:2.15
gson-api:2.13.1-139.v4569c2ef303f
handy-uri-templates-2-api:2.1.8-36.v85e4cb_234a_13
htmlpublisher:425
http_request:1.20
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:88.va_4187cb_eddf1
ivy:582.v35fb_da_0312f7
jackson2-api:2.18.3-402.v74c4eb_f122b_2
jakarta-activation-api:2.1.3-2
jakarta-mail-api:2.1.3-2
javadoc:327.vdfe586651ee0
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.19
jersey2-api:2.45-154.v4ded3dc34f81
jira:3.15
jjwt-api:0.11.5-120.v0268cf544b_89
joda-time-api:2.14.0-127.v7d9da_295a_d51
jquery:1.12.4-3
jquery3-api:3.7.1-3
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250107-125.v28b_a_ffa_eb_f01
json-path-api:2.9.0-148.v22a_7ffe323ce
jsoup:1.20.1-46.ve5f1416988c2
junit:1335.v6b_a_a_e18534e1
kubernetes:4340.v345364d31a_2a_
kubernetes-client-api:6.10.0-251.v556f5f100500
kubernetes-credentials:192.v4d5b_1c429d17
ldap:780.vcb_33c9a_e4332
leastload:62.vfa_8830902733
lockable-resources:1349.v8b_ccb_c5487f7
mailer:489.vd4b_25144138f
matrix-auth:3.2.6
matrix-project:849.v0cd64ed7e531
maven-plugin:3.26
metrics:4.2.30-471.v55fa_495f2b_f5
mina-sshd-api-common:2.15.0-161.vb_200831a_c15b_
mina-sshd-api-core:2.15.0-161.vb_200831a_c15b_
monitoring:2.5.0
next-build-number:66.v4b_4762172d53
oauth-credentials:0.657.v7d8dd90b_0382
okhttp-api:4.11.0-189.v976fa_d3379d6
openshift-client:1.1.0.424.v829cb_ccf8798
openshift-sync:1.1.0.818.v3883b_3b_df89a_
oss-symbols-api:324.v432cce4172ca_
pam-auth:1.12
parameterized-scheduler:285.ve611986d4c48
pipeline-build-step:567.vea_ce550ece97
pipeline-github-lib:65.v203688e7727e
pipeline-graph-analysis:237.v2b_75640ca_888
pipeline-graph-view:524.v313953a_7d4c6
pipeline-groovy-lib:752.vdddedf804e72
pipeline-input-step:517.vf8e782ee645c
pipeline-milestone-step:134.vdf60d179845f
pipeline-model-api:2.2255.v56a_15e805f12
pipeline-model-definition:2.2255.v56a_15e805f12
pipeline-model-extensions:2.2255.v56a_15e805f12
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2255.v56a_15e805f12
pipeline-stage-view:2.38
pipeline-utility-steps:2.19.0
plain-credentials:195.vb_906e9073dee
plugin-util-api:6.1.0
prism-api:1.30.0-1
prometheus:819.v50953a_c560dd
pubsub-light:1.19
resource-disposer:0.25
reverse-proxy-auth-plugin:238.v82ceca_8417a_6
run-condition:243.v3c3f94e46a_8b_
scm-api:704.v3ce5c542825a_
script-security:1373.vb_b_4a_a_c26fa_00
sidebar-link:2.4.2
simple-theme-plugin:211.v5424a_5510e47
snakeyaml-api:2.3-125.v4d77857a_b_402
sonar:2.18
sse-gateway:1.28
ssh-credentials:355.v9b_e5b_cde5003
ssh-slaves:3.1031.v72c6b_883b_869
sshd:3.353.v2b_d33c46e970
structs:343.vdcf37b_a_c81d5
support-core:1725.va_2a_9f06eed61
swarm:3.49
theme-manager:294.vf2c2fd783821
timestamper:1.28
token-macro:444.v52de7e9c573d
trilead-api:2.209.v0e69b_c43c245
variant:70.va_d9f17f859e0
versioncolumn:320.v6b_b_814ca_01f7
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1373.v7b_813f10efa_b_
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4106.v7a_8a_8176d450
workflow-durable-task-step:1405.v1fcd4a_d00096
workflow-job:1520.v56d65e3b_4566
workflow-multibranch:806.vb_b_688f609ee9
workflow-scm-step:437.v05a_f66b_e5ef8
workflow-step-api:700.v6e45cb_a_5a_a_21
workflow-support:968.v8f17397e87b_8
ws-cleanup:0.48
xunit:3.1.5
```
2 attachments
- [CPU Usage.png](https://issues.jenkins.io/secure/attachment/64378/CPU%20Usage.png)
> 
- [jenkins.log](https://issues.jenkins.io/secure/attachment/64377/jenkins.log)
Contributor guide
Assessment
This issue has not been assessed yet.