jenkinsci / jenkinsci/branch-api-plugin
[JENKINS-71717] plugin is too slow on NFS
- Dominant language
- Java
- Stars
- 39
- Forks
- 154
- PR merge metrics
- No merged PRs in 30d
Description
We have found a thread that is locked is this:
Computer.threadPoolForRemoting [#494987] / waiting for EC2 (DevSecOps Agent) - Linux (i-0b99db4829aeb5474) id=156631640" #13983096 daemon prio=5 os_prio=0 cpu=24150.51ms elapsed=8857.08s tid=0x00007f79d450c800 nid=0x2d447a in Object.wait() [0x00007f794d4b2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.18
/Native Method)
- waiting on
at hudson.remoting.Request.call(Request.java:177)
- waiting to re-lock in wait() <0x00000007aee98d98> (a hudson.remoting.UserRequest)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.FilePath.act(FilePath.java:1194)
at hudson.FilePath.list(FilePath.java:2063)
at hudson.FilePath.listDirectories(FilePath.java:2037)
at jenkins.branch.WorkspaceLocatorImpl$Collector.onOnline(WorkspaceLocatorImpl.java:612)
- locked <0x000000060d199a70> (a java.lang.String)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:748)
We don't know for sure why EC2s agent doesn't like this:
waiting for EC2 (DevSecOps Agent)
but we do know it is fine on local storage, but not NFS.
Perhaps this could be refactored in order to deal with a large amount of directories on NFS?
---
Originally reported by
douglasawh, imported from: plugin is too slow on NFS
Raw content of original issue
We have found a thread that is locked is this:
Computer.threadPoolForRemoting [#494987] / waiting for EC2 (DevSecOps Agent) - Linux (i-0b99db4829aeb5474) id=156631640" #13983096 daemon prio=5 os_prio=0 cpu=24150.51ms elapsed=8857.08s tid=0x00007f79d450c800 nid=0x2d447a in Object.wait() [0x00007f794d4b2000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.18/Native Method)
- waiting on <no object reference available>
at hudson.remoting.Request.call(Request.java:177)
- waiting to re-lock in wait() <0x00000007aee98d98> (a hudson.remoting.UserRequest)
at hudson.remoting.Channel.call(Channel.java:999)
at hudson.FilePath.act(FilePath.java:1194)
at hudson.FilePath.list(FilePath.java:2063)
at hudson.FilePath.listDirectories(FilePath.java:2037)
at jenkins.branch.WorkspaceLocatorImpl$Collector.onOnline(WorkspaceLocatorImpl.java:612)
- locked <0x000000060d199a70> (a java.lang.String)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:748)We don't know for sure why EC2s agent doesn't like this:
waiting for EC2 (DevSecOps Agent)but we do know it is fine on local storage, but not NFS.
Perhaps this could be refactored in order to deal with a large amount of directories on NFS?
environment
```
Jenkins 2.346.2
Branch API Plugin 2.1046.v0
```
2 attachments
- [image001.png](https://issues.jenkins.io/secure/attachment/60892/image001.png)
> 
- [image002.png](https://issues.jenkins.io/secure/attachment/60893/image002.png)
> 
Contributor guide
Research direction
Start at jenkins.branch.WorkspaceLocatorImpl$Collector.onOnline, especially line 612, and trace the calls through hudson.FilePath.listDirectories, FilePath.list, and SlaveComputer.setChannel. Compare the behavior on local storage and NFS using the supplied thread dump and attachments. Done means the workspace lookup no longer stalls or remains unacceptably slow when many directories are stored on NFS.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100