jenkinsci / jenkinsci/branch-api-plugin

[JENKINS-71717] plugin is too slow on NFS

Open
#777 1 comment 0 reactions 0 assignees View on GitHub
component:branch-api-plugin imported-jira-issue priority:major resolution:unresolved
Dominant language
Java
Stars
39
Forks
154
PR merge metrics
No merged PRs in 30d

Description

We have found a thread that is locked is this:

Computer.threadPoolForRemoting [#494987] / waiting for EC2 (DevSecOps Agent) - Linux (i-0b99db4829aeb5474) id=156631640" #13983096 daemon prio=5 os_prio=0 cpu=24150.51ms elapsed=8857.08s tid=0x00007f79d450c800 nid=0x2d447a in Object.wait()  [0x00007f794d4b2000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

        at java.lang.Object.wait(java.base@​11.0.18/Native Method)

        - waiting on

        at hudson.remoting.Request.call(Request.java:177)

        - waiting to re-lock in wait() <0x00000007aee98d98> (a hudson.remoting.UserRequest)

        at hudson.remoting.Channel.call(Channel.java:999)

        at hudson.FilePath.act(FilePath.java:1194)

        at hudson.FilePath.list(FilePath.java:2063)

        at hudson.FilePath.listDirectories(FilePath.java:2037)

        at jenkins.branch.WorkspaceLocatorImpl$Collector.onOnline(WorkspaceLocatorImpl.java:612)

        - locked <0x000000060d199a70> (a java.lang.String)

        at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:748)

We don't know for sure why EC2s agent doesn't like this:

waiting for EC2 (DevSecOps Agent)

but we do know it is fine on local storage, but not NFS.

What is interesting is this:

 

Perhaps this could be refactored in order to deal with a large amount of directories on NFS?

---
Originally reported by douglasawh, imported from: plugin is too slow on NFS


  • status: Open
  • priority: Major
  • component(s): branch-api-plugin
  • resolution: Unresolved
  • votes: 3
  • watchers: 1
  • imported: 20251211-141027

Raw content of original issue

We have found a thread that is locked is this:

Computer.threadPoolForRemoting [#494987] / waiting for EC2 (DevSecOps Agent) - Linux (i-0b99db4829aeb5474) id=156631640" #13983096 daemon prio=5 os_prio=0 cpu=24150.51ms elapsed=8857.08s tid=0x00007f79d450c800 nid=0x2d447a in Object.wait()  [0x00007f794d4b2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(java.base@11.0.18/Native Method)
        - waiting on <no object reference available>
        at hudson.remoting.Request.call(Request.java:177)
        - waiting to re-lock in wait() <0x00000007aee98d98> (a hudson.remoting.UserRequest)
        at hudson.remoting.Channel.call(Channel.java:999)
        at hudson.FilePath.act(FilePath.java:1194)
        at hudson.FilePath.list(FilePath.java:2063)
        at hudson.FilePath.listDirectories(FilePath.java:2037)
        at jenkins.branch.WorkspaceLocatorImpl$Collector.onOnline(WorkspaceLocatorImpl.java:612)
        - locked <0x000000060d199a70> (a java.lang.String)
        at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:748)

We don't know for sure why EC2s agent doesn't like this:
waiting for EC2 (DevSecOps Agent)

but we do know it is fine on local storage, but not NFS.

What is interesting is this:

 

Perhaps this could be refactored in order to deal with a large amount of directories on NFS?

environment

```
Jenkins 2.346.2

Branch API Plugin 2.1046.v0
```

2 attachments

- [image001.png](https://issues.jenkins.io/secure/attachment/60892/image001.png)
> ![image001.png](https://issues.jenkins.io/secure/attachment/60892/image001.png)
- [image002.png](https://issues.jenkins.io/secure/attachment/60893/image002.png)
> ![image002.png](https://issues.jenkins.io/secure/attachment/60893/image002.png)

Contributor guide

Open the contributing guide

Research direction

Start at jenkins.branch.WorkspaceLocatorImpl$Collector.onOnline, especially line 612, and trace the calls through hudson.FilePath.listDirectories, FilePath.list, and SlaveComputer.setChannel. Compare the behavior on local storage and NFS using the supplied thread dump and attachments. Done means the workspace lookup no longer stalls or remains unacceptably slow when many directories are stored on NFS.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.