[JENKINS-71979] Getting Timeout in processing large amount of directories in NFS

We are getting Timeout in the Jenkins Job using Branch API Plugin while processing large amount of directories stored in NFS.

Observation: During our testing, we observe that Branch API Plugin processing all the directories in sequential manner. When the amount of directories grown to large numbers, it starts taking more and more time.

Request for Improvement: Addition of Parallel Processing in Stages (using parallel Keyword) will help in processing directories in parallel. This will help in completing the processing in short span of time.

Test Environment Details:

1. Total Directory in NFS: 125K

2. Processing:

-> Iterating Directory

-> Printing Directory Name

3. Machine Specification for Jenkisn and NFS:

CPU: 4

Memory: 4 GB

OS: Rocky8

Result for Job1 (Doing Sequential Processing):

CPU Utilization: 20%

Execution Time: Job Failed, timeout reached - 5 minutes, Iterates only 28985 Directories

Result for Job2 (Doing Parallel Processing):

CPU Utilization: 50%

Execution Time: 3 Minutes 21 Seconds

Following changes done in the Job:

1. Jobs breaks into 5 Parallel Stages iterating 25K Directories each (using parallel Keyword)

2. Enabled "Pipeline speed/durability override" -> "Perforcemance-Optimized"

Attaching the code of both the Jobs for reference.

---
Originally reported by ashish_modi, imported from: Getting Timeout in processing large amount of directories in NFS

• status: Open
  
• priority: Major
  
• component(s): branch-api-plugin
  
• label(s): plugin
  
• resolution: Unresolved
  
• votes: 1
  
• watchers: 1
  
• imported: 20251211-141027
  

Raw content of original issue

We are getting Timeout in the Jenkins Job using Branch API Plugin while processing large amount of directories stored in NFS.


Observation: During our testing, we observe that Branch API Plugin processing all the directories in sequential manner. When the amount of directories grown to large numbers, it starts taking more and more time.


Request for Improvement: Addition of Parallel Processing in Stages (using parallel Keyword) will help in processing directories in parallel. This will help in completing the processing in short span of time.


Test Environment Details:


1. Total Directory in NFS: 125K


2. Processing:

-> Iterating Directory

-> Printing Directory Name


3. Machine Specification for Jenkisn and NFS:

CPU: 4

Memory: 4 GB

OS: Rocky8


Result for Job1 (Doing Sequential Processing):


CPU Utilization: 20%


Execution Time: Job Failed, timeout reached - 5 minutes, Iterates only 28985 Directories


Result for Job2 (Doing Parallel Processing):


CPU Utilization: 50%


Execution Time: 3 Minutes 21 Seconds


Following changes done in the Job:


1. Jobs breaks into 5 Parallel Stages iterating 25K Directories each (using parallel Keyword)


2. Enabled "Pipeline speed/durability override" -> "Perforcemance-Optimized"


Attaching the code of both the Jobs for reference.

• environment: Jenkins: 2.414.1 LTS, Branch API Plugin 2.1122

2 attachments

- [Groovy Script Parallel.txt](https://issues.jenkins.io/secure/attachment/61114/Groovy%20Script%20Parallel.txt)
- [Groovy Script Sequential.txt](https://issues.jenkins.io/secure/attachment/61113/Groovy%20Script%20Sequential.txt)