jenkinsci / jenkinsci/github-branch-source-plugin
[JENKINS-73559] Controller freezes due to ratelimiting
- Dominant language
- Java
- Stars
- 217
- Forks
- 398
- Avg merge
- 30m
- Merged PRs (30d)
- 1
Description
Issue Description
The GitHub Branch Source Plugin contains ratelimiting strategies to allow requests to GitHub to wait with execution if the ratelimit is (almost) exceeded to prevent them from failing - that is in general a good idea, however it can lead to plenty of Threads being locked at the same time, since Thread.sleep is being used. This then leads to the controller freezing.
This happens pretty easily especially with the anonymous rate limit (60 per hour per IP). This is for example used when clicking the "Validate" button in a pipeline configuration to check connectivity.
Once the ratelimit is now exceeded, you can click that "Validate" button a few times more each processing unit has a Thread waiting for the ratelimit to refresh - Jenkins stops being responsive.
When using the "Organisations folder" of this plugin, this happens even more likely since organisations make much more anonymous calls.
A real life example: Have an org-scan folder with a few dozen pipelines in it and the credentials expire - now there will be plenty of unauthenticated calls returning 401 - but counting towards the anonymous rate limit. This almost instantly freezes your controller. In our case, not even "sudo reboot" was able to kill the process.
Not sure, but this downstream locking logic may make this even worse - if one thread waits for this ratelimited-REST call to complete, this call blocks every other thread.
Sadly it is also not possible to simply ignore this ratelimit. Especially in environments behind a proxy (same outbound IP), this can cause Controllers to freeze pretty easily.
Reproduction steps
1) Clone the repository "github-branch-source-plugin"2
2) Launch a dev instance (mvn hpi:run)
3) Install the "Monitoring" (Java Melody) plugin, this helps us with debugging this issue.
4) Create a new multibranch pipeline and select "GitHub" as source type.
5) Put in any private repository, for example "https://github.com/github/github". Using credentials or not leads to the same result, but with no / invalid credentials, you can reproduce it much faster.
6) Click on "Validate" a few times until you see that you are being ratelimited in the console, i.e.:
2024-08-02 12:36:52.280+0000 [id=114] INFO o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: Jenkins-Imposed API Limiter: Current quota for Github API usage has 12 remaining (3 over budget). Next quota of 60 in 15 min. Sleeping until reset.
7) Open the Monitor at http://localhost:8080/jenkins/monitoring?.
Already now you will see that there is a sleeping Thread. There should be "1 current request"
Now, click the button a few more times until your CPU is completely occupied with sleeping threads.
Killing these threads via Java Melody will not help (if you even get through to that)
Possible solutions
I am really no expert with multithreading in Java, but this is what I thought of.
1) Use ScheduledExecutorService instead of Thread.sleep, however I do not have any experience with this so I would need a secondary opinion.
2) Allow to define a timeout for waiting for free rate limits
3) Allow the "Ignore Ratelimit" strategy for GitHub.com as well
---
Originally reported by meiswjn, imported from: Controller freezes due to ratelimiting
Raw content of original issue
Issue Description
The GitHub Branch Source Plugin contains ratelimiting strategies to allow requests to GitHub to wait with execution if the ratelimit is (almost) exceeded to prevent them from failing - that is in general a good idea, however it can lead to plenty of Threads being locked at the same time, since Thread.sleep is being used. This then leads to the controller freezing.
This happens pretty easily especially with the anonymous rate limit (60 per hour per IP). This is for example used when clicking the "Validate" button in a pipeline configuration to check connectivity.
Once the ratelimit is now exceeded, you can click that "Validate" button a few times more each processing unit has a Thread waiting for the ratelimit to refresh - Jenkins stops being responsive.
When using the "Organisations folder" of this plugin, this happens even more likely since organisations make much more anonymous calls.
A real life example: Have an org-scan folder with a few dozen pipelines in it and the credentials expire - now there will be plenty of unauthenticated calls returning 401 - but counting towards the anonymous rate limit. This almost instantly freezes your controller. In our case, not even "sudo reboot" was able to kill the process.
Not sure, but this downstream locking logic may make this even worse - if one thread waits for this ratelimited-REST call to complete, this call blocks every other thread.
Sadly it is also not possible to simply ignore this ratelimit. Especially in environments behind a proxy (same outbound IP), this can cause Controllers to freeze pretty easily.
Reproduction steps
1) Clone the repository "github-branch-source-plugin"2
2) Launch a dev instance (mvn hpi:run)
3) Install the "Monitoring" (Java Melody) plugin, this helps us with debugging this issue.
4) Create a new multibranch pipeline and select "GitHub" as source type.
5) Put in any private repository, for example "https://github.com/github/github". Using credentials or not leads to the same result, but with no / invalid credentials, you can reproduce it much faster.
6) Click on "Validate" a few times until you see that you are being ratelimited in the console, i.e.:
2024-08-02 12:36:52.280+0000 [id=114] INFO o.j.p.g.ApiRateLimitChecker$RateLimitCheckerAdapter#checkRateLimit: Jenkins-Imposed API Limiter: Current quota for Github API usage has 12 remaining (3 over budget). Next quota of 60 in 15 min. Sleeping until reset.
7) Open the Monitor at http://localhost:8080/jenkins/monitoring?.
Already now you will see that there is a sleeping Thread. There should be "1 current request"
Now, click the button a few more times until your CPU is completely occupied with sleeping threads.
Killing these threads via Java Melody will not help (if you even get through to that)
Possible solutions
I am really no expert with multithreading in Java, but this is what I thought of.
1) Use ScheduledExecutorService instead of Thread.sleep, however I do not have any experience with this so I would need a secondary opinion.
2) Allow to define a timeout for waiting for free rate limits
3) Allow the "Ignore Ratelimit" strategy for GitHub.com as well
environment
```
Jenkins: 2.414.3
OS: Windows 10 - 10.0 (just my local dev env to reproduce this)
Java: 11.0.21 - Amazon.com Inc. (OpenJDK 64-Bit Server VM)
---
All plugins below come by default with the dev instance of the plugin "github-branch-source", except for monitoring.
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
bootstrap5-api:5.3.2-3
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1128.v717130d4f816
caffeine-api:3.1.8-133.v17b_1ff2e0599
cloudbees-folder:6.858.v898218f3609d
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
configuration-as-code:1775.v810dc950b_514
credentials:1319.v7eb_51b_3a_c97b_
credentials-binding:657.v2b_19db_7d6e6d
display-url-api:2.200.vb_9327d658781
durable-task:555.v6802fe0f0b_82
font-awesome-api:6.5.1-2
git:5.2.1
git-client:4.7.0
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:999999-SNAPSHOT (private-1831e9c6-meiswjn)
gson-api:2.10.1-15.v0d99f670e0a_7
instance-identity:185.v303dc7c645f9
ionicons-api:70.v2959a_b_74e3cf
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-6
jaxb:2.3.9-1
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-33.v2527142f2e1d
mailer:470.vc91f60c5d8e2
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd
monitoring:1.99.0
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pipeline-groovy-lib:704.vc58b_8890a_384
plain-credentials:179.vc5cb_98f6db_38
plugin-util-api:3.8.0
prism-api:1.29.0-8
scm-api:676.v886669a_199a_a_
script-security:1335.vf07d9ce377a_e
snakeyaml-api:2.2-111.vc6598e30cc65
ssh-credentials:337.v395d2403ccd4
structs:337.v1b_04ea_4df7c8
token-macro:400.v35420b_922dcb_
trilead-api:2.142.v748523a_76693
variant:60.v7290fc0eb_b_cd
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1049.v257a_e6b_30fb_d
workflow-cps:3894.vd0f0248b_a_fc4
workflow-durable-task-step:1331.vc8c2fed35334
workflow-job:1385.vb_58b_86ea_fff1
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:896.v175a_a_9c5b_78f
```
1 attachment
- [image-2024-08-02-14-40-15-160.png](https://issues.jenkins.io/secure/attachment/63105/image-2024-08-02-14-40-15-160.png)
> 
Contributor guide
Assessment
This issue has not been assessed yet.