[JENKINS-73208] "Scan Repository" on a big one leads to huge amount of threads and kills our Jenkins

One of our repos under Github Organization has 4100 branches and 1100 pull requests.

The scanning is performed nightly, once it's running the amount of threads goes through the roof.

This happens right before "Finished branch indexing":

[Tue May 21 21:31:06 UTC 2024] Starting branch indexing...

...

[Tue May 21 22:09:16 UTC 2024] Finished branch indexing. Indexing took 38 min

Evaluating orphaned items in Pipeline Builder » 

It finished scanning at 22:09, here is the threads graph for that timeframe

Another multi-branch repo had 4330 branches and 3600 PRs.

I ran a manual scan, here's the result:

[Wed May 22 14:48:57 UTC 2024] Finished branch indexing. Indexing took 16 min

And the threads graph:





The Jenkins restarted at 14:50.

From docker logs:

2024-05-22 14:49:13.931+0000 [id=2668668]	WARNING	hudson.slaves.SlaveComputer#doJenkinsAgentJnlp: Agent "............." is connecting with the "-jnlpUrl" argument, which is deprecated. Use "-url" and "-name" instead, potentially also passing in "-webSocket", "-tunnel", and/or work directory options as needed.

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69800000, 16384, 0) failed; error='Not enough space' (errno=12)

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69700000, 16384, 0) failed; error='Not enough space' (errno=12)

[thread 301033 also had an error]#


# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory.

[thread 301034 also had an error]

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69600000, 16384, 0) failed; error='Not enough space' (errno=12)

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69500000, 16384, 0) failed; error='Not enough space' (errno=12)

[lots of similar messages]

This got to more than 21000 threads:

CheckMK

APP  17:49

Service PROBLEM notification

Host: jenkins.mycompany.domain (IP: jenkins.mycompany.domain)

Service: Number of threads

State: CRITICAL

Additional Info

CRIT - Count: 21608 threads (warn/crit at 3500 threads/4500 threads)(!!), Usage: 3.85%

Please take a look: @​cmkadmin

Check_MK notification: Wed May 22 14:49:15 GMT 2024

Please let me know if any other info needed.

---
Originally reported by eplodn1, imported from: "Scan Repository" on a big one leads to huge amount of threads and kills our Jenkins

• status: Open
  
• priority: Major
  
• component(s): branch-api-plugin
  
• resolution: Unresolved
  
• votes: 0
  
• watchers: 2
  
• imported: 20251211-141027
  

Raw content of original issue

One of our repos under Github Organization has 4100 branches and 1100 pull requests.

The scanning is performed nightly, once it's running the amount of threads goes through the roof.

  


This happens right before "Finished branch indexing":



[Tue May 21 21:31:06 UTC 2024] Starting branch indexing...

...

[Tue May 21 22:09:16 UTC 2024] Finished branch indexing. Indexing took 38 min

Evaluating orphaned items in Pipeline Builder » 





It finished scanning at 22:09, here is the threads graph for that timeframe

  


Another multi-branch repo had 4330 branches and 3600 PRs. 

I ran a manual scan, here's the result:



[Wed May 22 14:48:57 UTC 2024] Finished branch indexing. Indexing took 16 min





And the threads graph:

  

  

The Jenkins restarted at 14:50.


From docker logs:



2024-05-22 14:49:13.931+0000 [id=2668668]	WARNING	hudson.slaves.SlaveComputer#doJenkinsAgentJnlp: Agent "............." is connecting with the "-jnlpUrl" argument, which is deprecated. Use "-url" and "-name" instead, potentially also passing in "-webSocket", "-tunnel", and/or work directory options as needed.

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69800000, 16384, 0) failed; error='Not enough space' (errno=12)

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69700000, 16384, 0) failed; error='Not enough space' (errno=12)

[thread 301033 also had an error]#


# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory.

[thread 301034 also had an error]

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69600000, 16384, 0) failed; error='Not enough space' (errno=12)

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f8f69500000, 16384, 0) failed; error='Not enough space' (errno=12)

[lots of similar messages]





This got to more than 21000 threads:



CheckMK

APP  17:49

Service PROBLEM notification

Host: jenkins.mycompany.domain (IP: jenkins.mycompany.domain)

Service: Number of threads

State: CRITICAL

Additional Info

CRIT - Count: 21608 threads (warn/crit at 3500 threads/4500 threads)(!!), Usage: 3.85%

Please take a look: @cmkadmin

Check_MK notification: Wed May 22 14:49:15 GMT 2024





Please let me know if any other info needed.

environment

```
Jenkins: 2.440.1

OS: Linux - 5.15.0-106-generic

Java: 17.0.10 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

---

allure-jenkins-plugin:2.31.1

amazon-ecr:1.114.vfd22430621f5

analysis-model-api:12.3.3

ansicolor:1.0.4

antisamy-markup-formatter:162.v0e6ec0fcfcf6

apache-httpcomponents-client-4-api:4.5.14-208.v438351942757

apache-httpcomponents-client-5-api:5.3.1-1.0

artifact-manager-s3:845.848.ve968b_8576c1c

asm-api:9.7-33.v4d23ef79fcc8

authentication-tokens:1.53.v1c90fd9191a_b_

aws-bucket-credentials:1.0.0

aws-credentials:231.v08a_59f17d742

aws-global-configuration:130.v35b_7b_96f53c3

aws-java-sdk:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-api-gateway:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-autoscaling:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-cloudformation:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-cloudfront:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-codebuild:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-codedeploy:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-ec2:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-ecr:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-ecs:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-efs:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-elasticbeanstalk:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-elasticloadbalancingv2:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-iam:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-kinesis:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-lambda:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-logs:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-minimal:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-organizations:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-secretsmanager:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-sns:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-sqs:1.12.696-451.v0651a_da_9ca_ec

aws-java-sdk-ssm:1.12.696-451.v0651a_da_9ca_ec

basic-branch-build-strategies:81.v05e333931c7d

blueocean:1.27.12

blueocean-autofavorite:1.2.5

blueocean-bitbucket-pipeline:1.27.12

blueocean-commons:1.27.12

blueocean-config:1.27.12

blueocean-core-js:1.27.12

blueocean-dashboard:1.27.12

blueocean-display-url:2.4.2

blueocean-events:1.27.12

blueocean-git-pipeline:1.27.12

blueocean-github-pipeline:1.27.12

blueocean-i18n:1.27.12

blueocean-jira:1.27.12

blueocean-jwt:1.27.12

blueocean-personalization:1.27.12

blueocean-pipeline-api-impl:1.27.12

blueocean-pipeline-editor:1.27.12

blueocean-pipeline-scm-api:1.27.12

blueocean-rest:1.27.12

blueocean-rest-impl:1.27.12

blueocean-web:1.27.12

bootstrap5-api:5.3.3-1

bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9

branch-api:2.1169.va_f810c56e895

build-failure-analyzer:2.5.1

build-monitor-plugin:1.14-876.v2162064db_a_93

build-name-setter:2.4.2

build-time-blame:75.v44a_51cc97cdd-2.1.0

build-token-root:151.va_e52fe3215fc

build-token-trigger:1.0.0

buildtriggerbadge:251.vdf6ef853f3f5

built-on-column:1.4

caffeine-api:3.1.8-133.v17b_1ff2e0599

checks-api:2.2.0

cloud-stats:336.v788e4055508b_

cloudbees-bitbucket-branch-source:886.v44cf5e4ecec5

cloudbees-disk-usage-simple:203.v3f46a_7462b_1a_

cloudbees-folder:6.901.vb_4c7a_da_75da_3

cobertura:1.17

code-coverage-api:4.99.0

command-launcher:107.v773860566e2e

commons-httpclient3-api:3.1-3

commons-lang3-api:3.13.0-62.v7d18e55f51e2

commons-text-api:1.11.0-109.vfe16c66636eb_

compound-slaves:1.2

compress-artifacts:98.vb_20f3c77ddf7

conditional-buildstep:1.4.3

config-file-provider:973.vb_a_80ecb_9a_4d0

copyartifact:722.v0662a_9b_e22a_c

coverage:1.14.0

credentials:1337.v60b_d7b_c7b_c9f

credentials-binding:677.vdc9d38cb_254d

data-tables-api:2.0.7-1

description-setter:239.vd0a_6b_785f92d

display-url-api:2.204.vf6fddd8a_8b_e9

docker-commons:439.va_3cb_0a_6a_fb_29

docker-java-api:3.3.4-86.v39b_a_5ede342c

docker-plugin:1.6.1

docker-workflow:572.v950f58993843

durable-task:555.v6802fe0f0b_82

echarts-api:5.5.0-1

email-ext:1814.v404722f34263

envfile:43.v2644436e7d71

extended-choice-parameter:382.v5697b_32134e8

extended-read-permission:53.v6499940139e5

favorite:2.208.v91d65b_7792a_c

flaky-test-handler:1.2.3

font-awesome-api:6.5.2-1

forensics-api:2.4.0

generic-webhook-trigger:2.2.1

git:5.2.2

git-client:4.7.0

git-parameter:0.9.19

github:1.39.0

github-api:1.318-461.v7a_c09c9fa_d63

github-branch-source:1789.v5b_0c0cea_18c3

github-oauth:597.ve0c3480fcb_d0

github-scm-trait-notification-context:40.vfa_7f31a_b_d7f8

gradle:2.12

groovy:457.v99900cb_85593

gson-api:2.10.1-15.v0d99f670e0a_7

handy-uri-templates-2-api:2.1.8-30.v7e777411b_148

htmlpublisher:1.33

http_request:1.18

instance-identity:185.v303dc7c645f9

ionicons-api:74.v93d5eb_813d5f

jackson2-api:2.17.0-379.v02de8ec9f64c

jakarta-activation-api:2.1.3-1

jakarta-mail-api:2.1.3-1

javadoc:243.vb_b_503b_b_45537

javax-activation-api:1.2.0-6

javax-mail-api:1.6.2-9

jaxb:2.3.9-1

jdk-tool:73.vddf737284550

jenkins-design-language:1.27.12

jenkins-jira-issue-updater:1.18

jersey2-api:2.42-147.va_28a_44603b_d5

jira:3.13

jira-steps:2.0.165.v8846cf59f3db

jjwt-api:0.11.5-112.ve82dfb_224b_a_d

jnr-posix-api:3.1.19-2

job-dsl:1.87

job-restrictions:0.8

jobConfigHistory:1229.v3039470161a_d

joda-time-api:2.12.7-29.v5a_b_e3a_82269a_

jquery:1.12.4-1

jquery3-api:3.7.1-2

jsch:0.2.16-86.v42e010d9484b_

json-api:20240303-41.v94e11e6de726

json-path-api:2.9.0-58.v62e3e85b_a_655

junit:1265.v65b_14fa_f12f0

last-changes:2.7.11

ldap:725.v3cb_b_711b_1a_ef

locale:431.v3435fa_8f8445

lockable-resources:1255.vf48745da_35d0

log-parser:2.3.3

logstash:2.5.0218.v0a_ff8fefc12b_

mailer:472.vf7c289a_4b_420

mapdb-api:1.0.9-40.v58107308b_7a_7

material-theme:0.5.2-rc100.6121925fe229

matrix-auth:3.2.2

matrix-project:822.824.v14451b_c0fd42

maven-plugin:3.23

metrics:4.2.21-449.v6960d7c54c69

mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd

mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd

monitoring:1.98.0

next-build-number:1.8

okhttp-api:4.11.0-172.vda_da_1feeb_c6e

pam-auth:1.10

parameterized-scheduler:277.v61a_4b_a_49a_c5c

parameterized-trigger:806.vf6fff3e28c3e

performance:960.v701db_0d66e1b_

pipeline-aggregator-view:104.v94a_e5f6cdb_c3

pipeline-aws:1.45

pipeline-build-step:540.vb_e8849e1a_b_d8

pipeline-github:2.8-159.09e4403bc62f

pipeline-github-lib:42.v0739460cda_c4

pipeline-graph-analysis:216.vfd8b_ece330ca_

pipeline-graph-view:243.vc9e11fec486a_

pipeline-groovy-lib:710.v4b_94b_077a_808

pipeline-input-step:495.ve9c153f6067b_

pipeline-milestone-step:119.vdfdc43fc3b_9a_

pipeline-model-api:2.2198.v41dd8ef6dd56

pipeline-model-definition:2.2198.v41dd8ef6dd56

pipeline-model-extensions:2.2198.v41dd8ef6dd56

pipeline-rest-api:2.34

pipeline-stage-step:312.v8cd10304c27a_

pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56

pipeline-stage-view:2.34

pipeline-utility-steps:2.16.2

plain-credentials:182.v468b_97b_9dcb_8

plugin-util-api:4.1.0

postbuild-task:1.9

prism-api:1.29.0-15

project-description-setter:1.2

promoted-builds:945.v597f5c6a_d3fd

pubsub-light:1.18

read-only-configurations:1.10

rebuild:332.va_1ee476d8f6d

resource-disposer:0.23

role-strategy:717.v6a_69a_fe98974

run-condition:1.7

s3:466.vf5b_3db_8e3eb_2

schedule-build:572.v345ea_4a_a_d3fb_

scm-api:690.vfc8b_54395023

script-security:1336.vf33a_a_9863911

simple-theme-plugin:176.v39740c03a_a_f5

skip-notifications-trait:313.vd1337c8f8134

slack:715.v1cfed1b_9c63c

snakeyaml-api:2.2-111.vc6598e30cc65

sse-gateway:1.26

ssh:2.6.1

ssh-agent:367.vf9076cd4ee21

ssh-credentials:337.v395d2403ccd4

ssh-slaves:2.948.vb_8050d697fec

ssh-steps:2.0.68.va_d21a_12a_6476

sshd:3.322.v159e91f6a_550

structs:337.v1b_04ea_4df7c8

subversion:1256.vee91953217b_6

swarm:3.46

text-file-operations:1.3.2

text-finder:1.27

theme-manager:215.vc1ff18d67920

throttle-concurrents:2.14

timestamper:1.27

token-macro:400.v35420b_922dcb_

trilead-api:2.142.v748523a_76693

uno-choice:2.8.3

validating-string-parameter:183.v3748e79b_9737

variant:60.v7290fc0eb_b_cd

vdoo-vision:1.9

warnings-ng:11.3.0

workflow-aggregator:596.v8c21c963d92d

workflow-api:1311.v4250456a_e552

workflow-basic-steps:1058.vcb_fc1e3a_21a_9

workflow-cps:3894.3896.vca_2c931e7935

workflow-durable-task-step:1353.v1891a_b_01da_18

workflow-job:1400.v7fd111b_ec82f

workflow-multibranch:773.vc4fe1378f1d5

workflow-scm-step:427.v4ca_6512e7df1

workflow-step-api:657.v03b_e8115821b_

workflow-support:907.v6713a_ed8a_573

ws-cleanup:0.45

```

4 attachments

- [image (10).png](https://issues.jenkins.io/secure/attachment/62622/image%20%2810%29.png)
> 
- [image (11).png](https://issues.jenkins.io/secure/attachment/62621/image%20%2811%29.png)
> 
- [image (9).png](https://issues.jenkins.io/secure/attachment/62623/image%20%289%29.png)
> 
- [screenshot-1.png](https://issues.jenkins.io/secure/attachment/62624/screenshot-1.png)
> 

Start at the branch-api-plugin entry point for “Scan Repository” and follow the branch indexing period through the reported “Finished branch indexing” and orphaned-item evaluation messages. Reproduce with a repository containing thousands of branches and pull requests, then identify which operation creates the excess threads. Done should mean large scans complete without exhausting native memory or restarting Jenkins, with coverage for the observed behavior.

Written by the indexing model from the issue text.