jenkinsci / jenkinsci/junit-plugin

Memory leak related to `resultCache` in TestResultAction

Open
#652 28 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
87
Forks
351
Avg merge
1d 19h
Merged PRs (30d)
2

Description

### Jenkins and plugins versions report

Environment

```text

Jenkins: 2.462.1
OS: Linux - 6.8.4-3-pve
Java: 21.0.2 - S.A. (OpenJDK 64-Bit Server VM)
---
all-changes:1.5
analysis-model-api:12.4.0
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-110.v77252fb_d4da_5
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.119.v50285141b_7e1
badge:1.13
blueocean:1.27.14
blueocean-bitbucket-pipeline:1.27.14
blueocean-commons:1.27.14
blueocean-config:1.27.14
blueocean-core-js:1.27.14
blueocean-dashboard:1.27.14
blueocean-display-url:2.4.3
blueocean-events:1.27.14
blueocean-git-pipeline:1.27.14
blueocean-github-pipeline:1.27.14
blueocean-i18n:1.27.14
blueocean-jwt:1.27.14
blueocean-personalization:1.27.14
blueocean-pipeline-api-impl:1.27.14
blueocean-pipeline-editor:1.27.14
blueocean-pipeline-scm-api:1.27.14
blueocean-rest:1.27.14
blueocean-rest-impl:1.27.14
blueocean-web:1.27.14
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1178.v969d9eb_c728e
build-blocker-plugin:166.vc82fc20b_a_ed6
buildresult-trigger:0.18
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.0
cloudbees-bitbucket-branch-source:888.v8e6d479a_1730
cloudbees-folder:6.942.vb_43318a_156b_2
command-launcher:115.vd8b_301cc15d0
commons-compress-api:1.26.1-2
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.12.0-129.v99a_50df237f7
configuration-as-code:1850.va_a_8c31d3158b_
configuration-as-code-groovy:1.1
coverage:1.16.1
credentials:1371.vfee6b_095f0a_3
credentials-binding:681.vf91669a_32e45
cucumber-reports:5.8.3
dark-theme:479.v661b_1b_911c01
dashboard-view:2.517.v776a_b_811a_b_4e
data-tables-api:2.1.4-1
display-url-api:2.204.vf6fddd8a_8b_e9
durable-task:568.v8fb_5c57e8417
echarts-api:5.5.1-1
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1814.v404722f34263
envinject:2.919.v009a_a_1067cd0
envinject-api:1.199.v3ce31253ed13
extra-columns:1.26
favorite:2.221.v19ca_666b_62f5
flexible-publish:0.16.1
font-awesome-api:6.6.0-1
forensics-api:2.5.0
git:5.4.1
git-client:5.0.0
github:1.40.0
github-api:1.321-468.v6a_9f5f2d5a_7e
github-branch-source:1797.v86fdb_4d57d43
gitlab-plugin:1.8.1
groovy-postbuild:228.vcdb_cf7265066
gson-api:2.11.0-41.v019fcf6125dc
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.36
http_request:1.19
instance-identity:185.v303dc7c645f9
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:80.v8a_dee33ed6f0
jenkins-design-language:1.27.14
jersey2-api:2.44-151.v6df377fff741
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
job-dsl:1.88
jobConfigHistory:1241.v07634fa_18896
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-2
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1296.vb_f538b_c88630
mailer:472.vf7c289a_4b_420
matrix-auth:3.2.2
matrix-project:832.va_66e270d2946
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.13.2-125.v200281b_61d59
mina-sshd-api-core:2.13.2-125.v200281b_61d59
monitoring:1.99.0
nested-view:1.34
next-executions:327.v136ff959e97b_
oic-auth:4.324.vfd49d010926b_
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
parameterized-trigger:806.vf6fff3e28c3e
people-view:1.2
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-graph-view:332.vb_232ced67fa_9
pipeline-groovy-lib:730.ve57b_34648c63
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2214.vb_b_34b_2ea_9b_83
pipeline-model-definition:2.2214.vb_b_34b_2ea_9b_83
pipeline-model-extensions:2.2214.vb_b_34b_2ea_9b_83
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2214.vb_b_34b_2ea_9b_83
pipeline-stage-view:2.34
pipeline-utility-steps:2.17.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:4.1.0
postbuildscript:3.3.0-654.v67cf36130d78
prism-api:1.29.0-17
pubsub-light:1.18
rebuild:332.va_1ee476d8f6d
run-condition:1.7
scm-api:696.v778d637b_a_762
scoring-load-balancer:120.v5cc1a_77f0c09
script-security:999999-1341.va_2819b_414686-SNAPSHOT (private-a2819b41-dev)
sidebar-link:2.4.1
simple-theme-plugin:196.v96d9592f4efa_
snakeyaml-api:2.3-123.v13484c65210a_
sonar:2.17.2
sse-gateway:1.27
ssh-credentials:343.v884f71d78167
ssh-slaves:2.973.v0fa_8c0dea_f9f
sshd:3.330.vc866a_8389b_58
structs:338.v848422169819
test-results-analyzer:0.4.1
theme-manager:262.vc57ee4a_eda_5d
throttle-concurrents:2.14
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
variant:60.v7290fc0eb_b_cd
view-job-filters:382.vdf2d5e3f02f0
warnings-ng:11.4.1
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1336.vee415d95c521
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3953.v19f11da_8d2fa_
workflow-durable-task-step:1371.vb_7cec8f3b_95e
workflow-job:1436.vfa_244484591f
workflow-multibranch:795.ve0cb_1f45ca_9a_
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:920.v59f71ce16f04
```

### What Operating System are you using (both controller, and any agents involved in the problem)?

Controller: Ubuntu 20.04.6 LTS
Agents: Ubuntu 20.04.6 LTS

### Reproduction steps

Having the resultCache of TestResultAction enabled caused a slow memory build up for us. The problem seems to have started 3-4 weeks ago when we probably updated plugins along with a needed restart we had to do.

From then on, in a weeks time, more and more heap was being used until we were exhausting the 60GiB max heap.

### Expected Results

Memory usage should not steadily increase over time.

### Actual Results

Memory steadily building up until OOM
![image](https://github.com/user-attachments/assets/d93d1789-9df1-4eb0-894f-017338035004)
Note that the smaller hile in the middle was due to an unrelated controller restart.

Sadly I cannot recall (or know where to lookup) what changed around 3,4 weeks ago when the memory build up started to happen. But it looks like a recent change caused this.

### Anything else?

Clearing the resultCache (resultCache.clear()) freed up the 'lost' memory
![image](https://github.com/user-attachments/assets/a54b822a-2f61-4f07-95fa-c16af3f86cc8)

We disabled the cache via `hudson.tasks.junit.TestResultAction.RESULT_CACHE_ENABLED` and from now (26/09/2024) on , it looks like the memory build up is not happening anymore.

### Are you interested in contributing a fix?

Potentially yes, if can find some time, not an expert in Java though, so I might no be able to identify the source of the leak.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.