jenkinsci / jenkinsci/github-autostatus-plugin

Support on blocked InfluxDbNotifier

Open
#112 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
58
Forks
38
PR merge metrics
No merged PRs in 30d

Description

## Versions

Jenkins 2.289.2
github-autostatus:3.6.2
See other plugins in [plugins.txt](https://github.com/jenkinsci/github-autostatus-plugin/files/7305290/plugins.txt).

## Issue

I have tried to kill a build and now the thread finalizing the build is blocked on InfluxDbNotifier. Excerpt from `/threadDump`:

```java
"Handling POST /job/A/job/B/job/C/228/kill from : Jetty (winstone)-1375720" Id=1375720 Group=main BLOCKED on org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier@18915493 owned by "Running CpsFlowExecution[Owner[A/B/C/228:A/B/C #228]]" Id=1350112
at org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier.postData(InfluxDbNotifier.java:317)
- blocked on org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier@18915493
at org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier.notifyBuildStageStatus(InfluxDbNotifier.java:164)
at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager.lambda$notifyBuildStageStatus$0(BuildNotifierManager.java:131)
at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager$$Lambda$691/231652661.accept(Unknown Source)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager.notifyBuildStageStatus(BuildNotifierManager.java:130)
at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.updateBuildStatusForStage(BuildStatusAction.java:276)
at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.updateBuildStatusForStage(BuildStatusAction.java:288)
at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.lambda$close$1(BuildStatusAction.java:175)
at org.jenkinsci.plugins.githubautostatus.BuildStatusAction$$Lambda$706/2120780652.accept(Unknown Source)
at java.util.HashMap.forEach(HashMap.java:1289)
at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.close(BuildStatusAction.java:173)
at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.updateBuildStatusForJob(BuildStatusAction.java:298)
at org.jenkinsci.plugins.githubautostatus.BuildStatusJobListener.onCompleted(BuildStatusJobListener.java:83)
at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:617)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.doKill(WorkflowRun.java:486)
...
```

When trying

```groovy
Jenkins.instance.getItemByFullName("A/B/C")
.getBuildByNumber(228)
.finish(hudson.model.Result.ABORTED, new java.io.IOException("Aborting build"));
```

any subsequent attempts block the same:

```java
"Handling POST /script from : Jetty (winstone)-1437697" Id=1437697 Group=main BLOCKED on org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier@18915493 owned by "Running CpsFlowExecution[Owner[A/B/C/228:A/B/C #228]]" Id=1350112
at org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier.postData(InfluxDbNotifier.java:317)
- blocked on org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier@18915493
at org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier.notifyFinalBuildStatus(InfluxDbNotifier.java:198)
at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager.lambda$notifyFinalBuildStatus$1(BuildNotifierManager.java:143)
at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager$$Lambda$707/142026376.accept(Unknown Source)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager.notifyFinalBuildStatus(BuildNotifierManager.java:142)
at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.updateBuildStatusForJob(BuildStatusAction.java:299)
at org.jenkinsci.plugins.githubautostatus.BuildStatusJobListener.onCompleted(BuildStatusJobListener.java:83)
at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:233)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:133)
at Script1.run(Script1.groovy:3)
...
```

The owning thread seems to be stuck at reading from a socket (that probably doesn't reply anymore, due to network issues or timeouts):

```java
"Running CpsFlowExecution[Owner[A/B/C/228:A/B/C #228]]" Id=1350112 Group=main RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457)
at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165)
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109)
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1383)
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1291)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier.postData(InfluxDbNotifier.java:326)
- locked org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier@18915493
at org.jenkinsci.plugins.githubautostatus.notifiers.InfluxDbNotifier.notifyBuildStageStatus(InfluxDbNotifier.java:164)
at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager.lambda$notifyBuildStageStatus$0(BuildNotifierManager.java:131)
at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager$$Lambda$691/231652661.accept(Unknown Source)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.jenkinsci.plugins.githubautostatus.notifiers.BuildNotifierManager.notifyBuildStageStatus(BuildNotifierManager.java:130)
at org.jenkinsci.plugins.githubautostatus.BuildStatusAction.updateBuildStatusForStage(BuildStatusAction.java:276)
at org.jenkinsci.plugins.githubautostatus.GithubBuildStatusGraphListener.onNewHead(GithubBuildStatusGraphListener.java:132)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1473)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Number of locked synchronizers = 2
- java.util.concurrent.ThreadPoolExecutor$Worker@26857132
- java.util.concurrent.locks.ReentrantLock$NonfairSync@664d1da2
```

As the build hasn't finished for Jenkins any subsequent build attempts get queued.

## Questions

* So how do I get the Thread unstuck **without** restarting the Jenkins Controller instance? Is finding and closing the socket an option?
* I'm not sure how unlucky I got that I ended up in this situation. In Python when I'm in a comparable situation I normally put a timeout (and retries after some back-off with a counter) [around socket based operations](https://github.com/jenkinsci/github-autostatus-plugin/blob/github-autostatus-3.6.2/src/main/java/org/jenkinsci/plugins/githubautostatus/notifiers/InfluxDbNotifier.java#L326) so I can't get stuck there. Especially if a lock is acquired around the socket operation. My experience is that (our) networks aren't as reliable as we want them to be...

If you need more information, please let me know.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.