[JENKINS-68116] Slow processing of multi branch events

I'm seeing delays of about an hour from when a multi branch event is created in GitHub to when Jenkins actually processes it.

If I run the in the script console:

jenkins.scm.api.SCMEvent.executorService

I see:

java.util.concurrent.ScheduledThreadPoolExecutor@​8b0291b[Running, pool size = 10, active threads = 10, queued tasks = 2401, completed tasks = 12838]

Relevant code appears to be around:

https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/MultiBranchProject.java#L1179-L1198

and

https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/MultiBranchProject.java#L1385

Running:

Jenkins.get().getAllItems(jenkins.branch.MultiBranchProject.class).size()

Gives:

579 multi branch projects.

From what I can see in the log in

/var/jenkins_home/jenkins.branch.* logs

Events seem to be process in 0-1 seconds but based on Maths comparing events processed over an hour we were only processing 22 a minute, so some must be quite slow.

We only get to keep 15 minutes worth of logs as the file max size is set to 33 kilobytes for some reason:

https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/MultiBranchProject.java#L1138-L1142

Context:

Large organistion with 1.7K repositories, and an organisation webhook is pointing at Jenkins

Few ideas so far:

1. Throw more threads at it

https://github.com/jenkinsci/scm-api-plugin/blob/master/src/main/java/jenkins/scm/api/SCMEvent.java#L215

Is hardcoded to 10 threads at a time

2. Keep logs for longer

3. See if there's something specific that could be holding this up?

4. Global configuration to filter projects out of webhook processing? We have lots that don't need to be processed and will never be matched

---
Originally reported by timja, imported from: Slow processing of multi branch events

• status: Open
  
• priority: Minor
  
• component(s): branch-api-plugin, github-branch-source-plugin
  
• resolution: Unresolved
  
• votes: 1
  
• watchers: 6
  
• imported: 2025-12-02
  

Raw content of original issue

I'm seeing delays of about an hour from when a multi branch event is created in GitHub to when Jenkins actually processes it.


If I run the in the script console:



jenkins.scm.api.SCMEvent.executorService





I see:



java.util.concurrent.ScheduledThreadPoolExecutor@8b0291b[Running, pool size = 10, active threads = 10, queued tasks = 2401, completed tasks = 12838]





Relevant code appears to be around:

https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/MultiBranchProject.java#L1179-L1198

and


https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/MultiBranchProject.java#L1385


Running:



Jenkins.get().getAllItems(jenkins.branch.MultiBranchProject.class).size()





Gives:

579 multi branch projects.


From what I can see in the log in 

/var/jenkins_home/jenkins.branch.* logs


Events seem to be process in 0-1 seconds but based on Maths comparing events processed over an hour we were only processing 22 a minute, so some must be quite slow.


We only get to keep 15 minutes worth of logs as the file max size is set to 33 kilobytes for some reason:

 https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/MultiBranchProject.java#L1138-L1142


Context:

Large organistion with 1.7K repositories, and an organisation webhook is pointing at Jenkins


Few ideas so far:

1. Throw more threads at it


https://github.com/jenkinsci/scm-api-plugin/blob/master/src/main/java/jenkins/scm/api/SCMEvent.java#L215


Is hardcoded to 10 threads at a time


2. Keep logs for longer


3. See if there's something specific that could be holding this up?


4. Global configuration  to filter projects out of webhook processing? We have lots that don't need to be processed and will never be matched

environment

```
Jenkins: 2.321

OS: Linux - 5.4.0-1073-azure

---

ace-editor:1.1

ansicolor:1.0.1

antisamy-markup-formatter:2.5

apache-httpcomponents-client-4-api:4.5.13-1.0

authentication-tokens:1.4

azure-ad:185.v3b416408dcb1

azure-artifact-manager:97.v074e1332e88d

azure-credentials:198.vf9c2fdfde55c

azure-keyvault:131.v867845ef6ae9

azure-sdk:70.v63f6a95999a7

azure-vm-agents:799.va4c741108611

basic-branch-build-strategies:1.3.2

blueocean:1.25.2

blueocean-autofavorite:1.2.4

blueocean-bitbucket-pipeline:1.25.2

blueocean-commons:1.25.2

blueocean-config:1.25.2

blueocean-core-js:1.25.2

blueocean-dashboard:1.25.2

blueocean-display-url:2.4.1

blueocean-events:1.25.2

blueocean-git-pipeline:1.25.2

blueocean-github-pipeline:1.25.2

blueocean-i18n:1.25.2

blueocean-jwt:1.25.2

blueocean-personalization:1.25.2

blueocean-pipeline-api-impl:1.25.2

blueocean-pipeline-editor:1.25.2

blueocean-pipeline-scm-api:1.25.2

blueocean-rest:1.25.2

blueocean-rest-impl:1.25.2

blueocean-web:1.25.2

bootstrap4-api:4.6.0-3

bootstrap5-api:5.1.3-3

bouncycastle-api:2.25

branch-api:2.7.0

build-monitor-plugin:1.13+build.202111220004

caffeine-api:2.9.2-29.v717aac953ff3

checks-api:1.7.2

cloud-stats:0.27

cloudbees-bitbucket-branch-source:726.vb0c1ea6c9336

cloudbees-disk-usage-simple:0.10

cloudbees-folder:6.16

cloudbees-jenkins-advisor:3.3.2

command-launcher:1.6

configuration-as-code:1.54

copyartifact:1.46.2

credentials:2.6.2

credentials-binding:1.27

dark-theme:155.v497c78bbdbb3

display-url-api:2.3.5

docker-commons:1.17

docker-workflow:1.26

durable-task:493.v195aefbb0ff2

echarts-api:5.2.2-1

extended-read-permission:3.2

favorite:2.3.3

font-awesome-api:5.15.4-3

gatling:1.3.0

git:4.10.0

git-client:3.10.0

git-server:1.10

github:1.34.1

github-api:1.133

github-branch-source:2.11.3

github-checks:1.0.13

github-scm-trait-notification-context:1.1

handlebars:3.0.8

handy-uri-templates-2-api:2.1.8-1.0

htmlpublisher:1.28

http_request:1.12

jackson2-api:2.13.0-230.v59243c64b0a5

jacoco:3.3.0

javadoc:1.6

jaxb:2.3.0.1

jdk-tool:1.5

jenkins-design-language:1.25.2

jjwt-api:0.11.2-9.c8b45b8bb173

job-dsl:1.78.1

jquery3-api:3.6.0-2

jsch:0.1.55.2

junit:1.53

kubernetes:1.30.10

kubernetes-client-api:5.4.1

kubernetes-credentials:0.9.0

lockable-resources:2.12

mailer:1.34

matrix-auth:2.6.8

matrix-project:1.19

maven-plugin:3.15.1

metrics:4.0.2.8

momentjs:1.1.1

monitoring:1.88.0

okhttp-api:4.9.2-20211102

pipeline-build-step:2.15

pipeline-github-lib:1.0

pipeline-graph-analysis:1.11

pipeline-graph-view:46.vedbf23da719f

pipeline-input-step:2.12

pipeline-milestone-step:1.3.2

pipeline-model-api:1.9.3

pipeline-model-definition:1.9.3

pipeline-model-extensions:1.9.3

pipeline-rest-api:2.19

pipeline-stage-step:2.5

pipeline-stage-tags-metadata:1.9.3

pipeline-stage-view:2.19

pipeline-utility-steps:2.10.0

plain-credentials:1.7

plugin-util-api:2.5.1

popper-api:1.16.1-2

popper2-api:2.10.2-1

prometheus:2.0.10

pubsub-light:1.16

run-condition:1.5

saml:2.0.9

sauce-ondemand:1.197

scm-api:2.6.5

script-security:1.78

slack:2.48

snakeyaml-api:1.29.1

sonar:2.14

sse-gateway:1.24

ssh-credentials:1.19

sshd:3.1.0

structs:1.23

support-core:2.79

theme-manager:0.6

timestamper:1.14

token-macro:267.vcdaea6462991

trilead-api:1.0.13

variant:1.4

windows-azure-storage:365.vf41653c43b01

workflow-aggregator:2.6

workflow-api:2.47

workflow-basic-steps:2.24

workflow-cps:2633.v6baeedc13805

workflow-cps-global-lib:548.v9085a486966a

workflow-durable-task-step:1097.veac1aacfbda8

workflow-job:2.42

workflow-multibranch:2.26

workflow-scm-step:2.13

workflow-step-api:2.24

workflow-support:3.8
```

18 attachments

- [events-trace-90s.txt](https://issues.jenkins.io/secure/attachment/57581/events-trace-90s.txt)
- [events-trace.txt](https://issues.jenkins.io/secure/attachment/57580/events-trace.txt)
- [github-api-get-repos-sorted.txt](https://issues.jenkins.io/secure/attachment/57593/github-api-get-repos-sorted.txt)
- [over-1-second.txt](https://issues.jenkins.io/secure/attachment/57562/over-1-second.txt)
- [over-10-seconds-29-3.txt](https://issues.jenkins.io/secure/attachment/57578/over-10-seconds-29-3.txt)
- [over-10-seconds.txt](https://issues.jenkins.io/secure/attachment/57563/over-10-seconds.txt)
- [threads.6.20220325093536.txt](https://issues.jenkins.io/secure/attachment/57544/threads.6.20220325093536.txt)
- [threads.6.20220325093550.txt](https://issues.jenkins.io/secure/attachment/57545/threads.6.20220325093550.txt)
- [threads.6.20220325093602.txt](https://issues.jenkins.io/secure/attachment/57546/threads.6.20220325093602.txt)
- [threads.6.20220325093615.txt](https://issues.jenkins.io/secure/attachment/57547/threads.6.20220325093615.txt)
- [threads.6.20220325093629.txt](https://issues.jenkins.io/secure/attachment/57548/threads.6.20220325093629.txt)
- [threads.6.20220325093641.txt](https://issues.jenkins.io/secure/attachment/57549/threads.6.20220325093641.txt)
- [threads.6.20220325093655.txt](https://issues.jenkins.io/secure/attachment/57550/threads.6.20220325093655.txt)
- [threads.6.20220325093706.txt](https://issues.jenkins.io/secure/attachment/57551/threads.6.20220325093706.txt)
- [threads.6.20220325093720.txt](https://issues.jenkins.io/secure/attachment/57552/threads.6.20220325093720.txt)
- [threads.6.20220325093732.txt](https://issues.jenkins.io/secure/attachment/57553/threads.6.20220325093732.txt)
- [threads.6.20220325093745.txt](https://issues.jenkins.io/secure/attachment/57554/threads.6.20220325093745.txt)
- [threads.6.20220325093759.txt](https://issues.jenkins.io/secure/attachment/57555/threads.6.20220325093759.txt)