jenkinsci / jenkinsci/gerrit-trigger-plugin
[JENKINS-52636] Gerrit triggered jobs getting delayed
- Dominant language
- Java
- Stars
- 226
- Forks
- 289
- PR merge metrics
- No merged PRs in 30d
Description
We are experiencing a delay in Gerrit triggered Jobs in our Jenkins Jobs.
attached is the stack trace of the blocking threads.
Jul 18, 2018 5:07:57 PM com.sonymobile.tools.gerrit.gerritevents.GerritHandler checkQueueSize
WARNING: The Gerrit incoming events queue contains 28095 items! Something might be stuck, or your system can't process the commands fast enough. Try to increase the number of receiving worker threads. Current thread-pool size: 30
Jul 18, 2018 6:54:37 PM com.sonymobile.tools.gerrit.gerritevents.GerritJsonEventFactory getEvent
FINE: Constructor with JSONObject as parameter missing, trying default constructor.
java.lang.NoSuchMethodException: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated.(net.sf.json.JSONObject)
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getConstructor(Class.java:1825)
at com.sonymobile.tools.gerrit.gerritevents.GerritJsonEventFactory.getEvent(GerritJsonEventFactory.java:69)
at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractJsonObjectWork.perform(AbstractJsonObjectWork.java:69)
at com.sonymobile.tools.gerrit.gerritevents.workers.StreamEventsStringWork.perform(StreamEventsStringWork.java:67)
at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66)
at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)
---
Originally reported by eattsma, imported from: Gerrit triggered jobs getting delayed
scoheb
Raw content of original issue
We are experiencing a delay in Gerrit triggered Jobs in our Jenkins Jobs.
attached is the stack trace of the blocking threads.
Jul 18, 2018 5:07:57 PM com.sonymobile.tools.gerrit.gerritevents.GerritHandler checkQueueSize
WARNING: The Gerrit incoming events queue contains 28095 items! Something might be stuck, or your system can't process the commands fast enough. Try to increase the number of receiving worker threads. Current thread-pool size: 30
Jul 18, 2018 6:54:37 PM com.sonymobile.tools.gerrit.gerritevents.GerritJsonEventFactory getEvent
FINE: Constructor with JSONObject as parameter missing, trying default constructor.
java.lang.NoSuchMethodException: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated.<init>(net.sf.json.JSONObject)
at java.lang.Class.getConstructor0(Class.java:3082)
at java.lang.Class.getConstructor(Class.java:1825)
at com.sonymobile.tools.gerrit.gerritevents.GerritJsonEventFactory.getEvent(GerritJsonEventFactory.java:69)
at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractJsonObjectWork.perform(AbstractJsonObjectWork.java:69)
at com.sonymobile.tools.gerrit.gerritevents.workers.StreamEventsStringWork.perform(StreamEventsStringWork.java:67)
at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66)
at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)
environment
```
Operating System :
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 6.8 (Santiago)
Release: 6.8
Codename: Santiago
JRE/JDK vendors and versions : 1.8.0_45
Jenkins versions : 2.46.2
Gerrit grigger plugin versions : 2.27.5
pipeline plugin version : 2.5
script-security plugin version : 1.44
running Jenkins directly or in a container : Jenkins is running on a Virtual machine on tomcat.
How you installed Jenkins : jenkins.war file to install Jenkins.
how you're launching any involved slave nodes : yes, we have slaves using ssh.
Your web browser : Crome Version 67.0.3396.99 (Official Build) (64-bit)
```
3 attachments
- [fem019-eiffel003_td__1.txt](https://issues.jenkins.io/secure/attachment/43507/fem019-eiffel003_td__1.txt)
- [fem019-eiffel003_td__3.txt](https://issues.jenkins.io/secure/attachment/43508/fem019-eiffel003_td__3.txt)
- [plugins.txt](https://issues.jenkins.io/secure/attachment/43428/plugins.txt)
Contributor guide
Research direction
Start with GerritJsonEventFactory.java:69 and trace the event flow through AbstractJsonObjectWork.java, StreamEventsStringWork.java, and EventThread.java using the supplied stack trace and queue warning. Compare RefUpdated event handling with the attached logs; done means identifying and correcting the cause of the growing incoming-event queue so Gerrit-triggered jobs are no longer delayed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, ci-cd
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100