jenkinsci / jenkinsci/github-branch-source-plugin

[JENKINS-68321] SCMEvent threads waiting on rate limit when rate limit isn't close to being hit

Open
#1,395 18 comments 0 reactions 0 assignees View on GitHub
component:github-branch-source-plugin imported-jira-issue priority:major resolution:unresolved
Dominant language
Java
Stars
217
Forks
398
Avg merge
30m
Merged PRs (30d)
1

Description

Our organization is experiencing large delays when jenkins processes webhooks (on the order of minutes up to hours). When looking at a threadDump, the following is observed in all SCMEvent threads:

class 

org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl

 Wed Apr 20 14:18:16 EDT 2022 / SCMEvent [#4]"class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Wed Apr 20 14:18:16 EDT 2022 / SCMEvent [#4]" Id=2608 Group=main TIMED_WAITING

	at java.lang.Thread.sleep(Native Method)

	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:325)

	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.checkRateLimit(ApiRateLimitChecker.java:261)

	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$RateLimitCheckerAdapter.checkRateLimit(ApiRateLimitChecker.java:242)

	at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:128)

	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:383)

	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:355)

	at org.kohsuke.github.Requester.fetch(Requester.java:76)

	at org.kohsuke.github.GHRepository.read(GHRepository.java:132)

	at org.kohsuke.github.GHPerson.getRepository(GHPerson.java:146)

	at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSource(GitHubSCMNavigator.java:1389)

	at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:926)

	at jenkins.scm.api.SCMNavigator.visitSources(SCMNavigator.java:221)

	at jenkins.branch.OrganizationFolder$SCMEventListenerImpl.onSCMHeadEvent(OrganizationFolder.java:1049)

	at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246)

	at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229)

	at jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:505)

	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)

	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

	at java.util.concurrent.FutureTask.run(FutureTask.java:266)

	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

	at java.lang.Thread.run(Thread.java:750)



	Number of locked synchronizers = 1

	- java.util.concurrent.ThreadPoolExecutor$Worker@​594bc15a

However, the rate limit for the service account has not come close to 0. The minimum observed is 3000 out of 5000 remaining. This is observed on dashboards as well as when testing the connection from the jenkins UI.

We are using public GitHub. The rate limiting strategy is set to "Throttle at/near rate limit". It used to be set to "Normalize API requests", but this exacerbated the problem. 

Notably, the following is seen in the github branch source logs:

2022-04-20 20:09:02.440+0000 [id=247607]        INFO    o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings.

2022-04-20 20:09:02.512+0000 [id=247609]        INFO    o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings.

2022-04-20 20:09:02.512+0000 [id=247608]        INFO    o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings.

The following is also seen every few seconds in the github branch source logs:

2022-04-20 20:09:21.187+0000 [id=247232]        FINE    jenkins.scm.api.SCMSource#defaultListener: Connecting to https://api.github.com using REDACTED

Let me know if any other information would be helpful.

---
Originally reported by gduffy, imported from: SCMEvent threads waiting on rate limit when rate limit isn't close to being hit


  • status: Open
  • priority: Major
  • component(s): github-branch-source-plugin
  • resolution: Unresolved
  • votes: 2
  • watchers: 7
  • imported: 2025-12-02

Raw content of original issue

Our organization is experiencing large delays when jenkins processes webhooks (on the order of minutes up to hours). When looking at a threadDump, the following is observed in all SCMEvent threads:




class 

org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl

 Wed Apr 20 14:18:16 EDT 2022 / SCMEvent [#4]"class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Wed Apr 20 14:18:16 EDT 2022 / SCMEvent [#4]" Id=2608 Group=main TIMED_WAITING

	at java.lang.Thread.sleep(Native Method)

	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:325)

	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.checkRateLimit(ApiRateLimitChecker.java:261)

	at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$RateLimitCheckerAdapter.checkRateLimit(ApiRateLimitChecker.java:242)

	at org.kohsuke.github.GitHubRateLimitChecker.checkRateLimit(GitHubRateLimitChecker.java:128)

	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:383)

	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:355)

	at org.kohsuke.github.Requester.fetch(Requester.java:76)

	at org.kohsuke.github.GHRepository.read(GHRepository.java:132)

	at org.kohsuke.github.GHPerson.getRepository(GHPerson.java:146)

	at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSource(GitHubSCMNavigator.java:1389)

	at org.jenkinsci.plugins.github_branch_source.GitHubSCMNavigator.visitSources(GitHubSCMNavigator.java:926)

	at jenkins.scm.api.SCMNavigator.visitSources(SCMNavigator.java:221)

	at jenkins.branch.OrganizationFolder$SCMEventListenerImpl.onSCMHeadEvent(OrganizationFolder.java:1049)

	at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246)

	at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229)

	at jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:505)

	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)

	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

	at java.util.concurrent.FutureTask.run(FutureTask.java:266)

	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

	at java.lang.Thread.run(Thread.java:750)



	Number of locked synchronizers = 1

	- java.util.concurrent.ThreadPoolExecutor$Worker@594bc15a





However, the rate limit for the service account has not come close to 0. The minimum observed is 3000 out of 5000 remaining. This is observed on dashboards as well as when testing the connection from the jenkins UI.



We are using public GitHub. The rate limiting strategy is set to "Throttle at/near rate limit". It used to be set to "Normalize API requests", but this exacerbated the problem. 



Notably, the following is seen in the github branch source logs:




2022-04-20 20:09:02.440+0000 [id=247607]        INFO    o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings.

2022-04-20 20:09:02.512+0000 [id=247609]        INFO    o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings.

2022-04-20 20:09:02.512+0000 [id=247608]        INFO    o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: LocalChecker for rate limit was not set for this thread. Configured using system settings. 




The following is also seen every few seconds in the github branch source logs:




2022-04-20 20:09:21.187+0000 [id=247232]        FINE    jenkins.scm.api.SCMSource#defaultListener: Connecting to https://api.github.com using REDACTED 




Let me know if any other information would be helpful.

  • environment: GitHub Branch Source Plugin version 2.11.4

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.