jenkinsci / jenkinsci/branch-api-plugin

[JENKINS-75144] One unauthenticated OrganizationFolder blocks threads for all OrganizationFolders

Open
#797 1 comment 0 reactions 0 assignees View on GitHub
component:branch-api-plugin imported-jira-issue priority:major resolution:unresolved
Dominant language
Java
Stars
39
Forks
154
PR merge metrics
No merged PRs in 30d

Description

When Jenkins receives an incoming webhook that should trigger jobs in OrganizationFolders, it will iterate synchronously over every folder and visit its sources to check for triggerable jobs: https://github.com/jenkinsci/branch-api-plugin/blob/d9f35001c95c40294f9b3e9065f68cb9f2be0351/src/main/java/jenkins/branch/OrganizationFolder.java#L992 (see also line 1048).

When even one OrganizationFolder is misconfigured in a way that it targets e.g. a GitHub Organization, but has no credentials specified (see screenshot), it will use anonymous API calls (for GitHub the quota is 60 / hour). When this quota is reached, the corresponding plugin (in this case GitHub Branch Source Plugin) will pause the thread until quota is replenished: https://github.com/jenkinsci/github-branch-source-plugin/blob/088b5f22c7680d6e4d29ce39542d32f9c2b2998f/src/main/java/org/jenkinsci/plugins/github_branch_source/ApiRateLimitChecker.java#L266

Eventually, all available threads will get blocked by this sleep. When that happens, webhooks are no longer being processed for any folder, even the properly configured ones. This can be observed in the thread monitoring page of Jenkins as N number of threads in TIMED_WAIT state where N equals jenkins.scm.api.SCMEvent.EVENT_THREAD_POOL_SIZE.

In my organization's Jenkins, which has ~20 active OrganizationFolders and handles several webhooks per minute, one such folder disabled webhooks for all folders within an hour of rebooting the instance, hence I'm marking this as Major.

I believe the branch-api-plugin should handle the OrganizationFolder iteration asynchronously, or have some kind of a fail-safe (e.g. timeout) in case handling a Folder blocks the thread. This can happen with other SCM vendors (and corresponding plugins), hence I am opening this issue with this component.


Steps to reproduce



  • Configure a GitHub Organization Folder with proper credentials, set up a webhook and verify that it works properly

  • Add another GitHub Organization Folder for the same organization, but leave credentials empty

  • Trigger webhook ~100 times within 1 hour

    • 60 times to saturate anonymous GitHub API quota

    • ~10 times to saturate thread pool of default size



---
Originally reported by waked, imported from: One unauthenticated OrganizationFolder blocks threads for all OrganizationFolders


  • status: Open
  • priority: Major
  • component(s): branch-api-plugin
  • resolution: Unresolved
  • votes: 0
  • watchers: 1
  • imported: 20251211-141027

Raw content of original issue

When Jenkins receives an incoming webhook that should trigger jobs in OrganizationFolders, it will iterate synchronously over every folder and visit its sources to check for triggerable jobs: https://github.com/jenkinsci/branch-api-plugin/blob/d9f35001c95c40294f9b3e9065f68cb9f2be0351/src/main/java/jenkins/branch/OrganizationFolder.java#L992 (see also line 1048).

When even one OrganizationFolder is misconfigured in a way that it targets e.g. a GitHub Organization, but has no credentials specified (see screenshot), it will use anonymous API calls (for GitHub the quota is 60 / hour). When this quota is reached, the corresponding plugin (in this case GitHub Branch Source Plugin) will pause the thread until quota is replenished: https://github.com/jenkinsci/github-branch-source-plugin/blob/088b5f22c7680d6e4d29ce39542d32f9c2b2998f/src/main/java/org/jenkinsci/plugins/github_branch_source/ApiRateLimitChecker.java#L266

Eventually, all available threads will get blocked by this sleep. When that happens, webhooks are no longer being processed for any folder, even the properly configured ones. This can be observed in the thread monitoring page of Jenkins as N number of threads in TIMED_WAIT state where N equals jenkins.scm.api.SCMEvent.EVENT_THREAD_POOL_SIZE.

In my organization's Jenkins, which has ~20 active OrganizationFolders and handles several webhooks per minute, one such folder disabled webhooks for all folders within an hour of rebooting the instance, hence I'm marking this as Major.

I believe the branch-api-plugin should handle the OrganizationFolder iteration asynchronously, or have some kind of a fail-safe (e.g. timeout) in case handling a Folder blocks the thread. This can happen with other SCM vendors (and corresponding plugins), hence I am opening this issue with this component.


Steps to reproduce



  • Configure a GitHub Organization Folder with proper credentials, set up a webhook and verify that it works properly

  • Add another GitHub Organization Folder for the same organization, but leave credentials empty

  • Trigger webhook ~100 times within 1 hour

    • 60 times to saturate anonymous GitHub API quota

    • ~10 times to saturate thread pool of default size




environment

```
Jenkins 2.462.3

Branch API Plugin 2.1200.v4b_a_3da_2eb_db_4

GitHub Branch Source Plugin 1807.v50351eb_7dd13
```

1 attachment

- [Screenshot 2025-01-15 at 10.26.09.png](https://issues.jenkins.io/secure/attachment/63808/Screenshot%202025-01-15%20at%2010.26.09.png)
> ![Screenshot 2025-01-15 at 10.26.09.png](https://issues.jenkins.io/secure/attachment/63808/Screenshot%202025-01-15%20at%2010.26.09.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.