nextflow-io / nextflow-io/nextflow
PBSPro executor spamming PBS server with requests
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Expected behavior and actual behavior
Expected: on a cluster with a large amount of jobs running at any given time (independent of Nextflow jobs) Nextflow's PBS/PBS Pro status polling should query job state in a way that doesn't add meaningful load to pbs_server.
Actual: The PBSPro's executor's polling runs qstat -f, which returns the full, verbose per-job attribute listing for all jobs on the cluster. On a shared cluster where thousands of jobs are queued/running across at any given time, qstat -f effectively "spams" the server with expensive status calls causing the PBS server to verge on crashing, thereby making the cluster unusable due to any PBS commands not responding.
Source issue lines:
Functions in : https://github.com/nextflow-io/nextflow/blob/2e13110f20f4afc9e54ca8a7d013b952772175b5/modules/nextflow/src/main/groovy/nextflow/executor/PbsProExecutor.groovy#L102
Steps to reproduce the problem:
On a PBS/PBS Pro cluster that regularly carries several thousand queued/running jobs across all users,
Configure process.executor = 'pbs' (or pbspro) and run any Nextflow pipeline.
Observe the status-polling command Nextflow issues at each poll interval - confirm it invokes qstat -f.
On the PBS server side, measure qstat -f response time/server load when the cluster-wide job count is in the thousands, versus a lightly loaded cluster. (grep "Type 19" /var/spool/pbs/server_logs/{log file})
Observe that poll latency and server load scale with total cluster job count, not with the number of jobs belonging to the Nextflow run being monitored.
From our PBS server where the issue was seen in production:
grep "Type 19" /var/spool/pbs/server_logs/20260822 | wc -l
50390
We are a high throughput system so for visibility, this is the number of jobs we have running including subjobs:
# qstat -t | wc -l
289618
qstat outputs before and after killing all nexflow processes across the entire cluster:
$ time qstat -q
server: pbs-7
Queue Memory CPU Time Walltime Node Run Que Lm State
---------------- ------ -------- -------- ---- ----- ----- ---- -----
v1_gpu72 920gb -- 72:00:00 1 75 489 -- E R
v1_small24 128gb -- 24:00:00 1 478 23520 -- E R
v1_small24a 128gb -- 24:00:00 1 926 194 -- E R
v1_small72 128gb -- 72:00:00 1 196 248 -- E R
v1_small72a 128gb -- 72:00:00 1 2524 695 -- E R
v1_medium24 450gb -- 24:00:00 1 25 201 -- E R
v1_medium24a 450gb -- 24:00:00 1 36 9 -- E R
v1_medium72 450gb -- 72:00:00 1 104 34 -- E R
v1_medium72a 450gb -- 72:00:00 1 228 28 -- E R
v1_large24 920gb -- 24:00:00 1 17 103 -- E R
v1_large24a 920gb -- 24:00:00 1 7 55 -- E R
v1_large72 920gb -- 72:00:00 1 53 31 -- E R
v1_large72a 920gb -- 72:00:00 1 65 162 -- E R
v1_largemem72 4000g -- 72:00:00 1 15 22 -- E R
v1_capability24 2048g -- 24:00:00 4 12 16 -- E R
v1_capability48 2048g -- 48:00:00 4 11 26 -- E R
v1_jupyter 64gb -- 08:00:00 1 18 12 -- E R
cx 4000g -- 72:00:00 4 0 1 -- E R
system_queue -- -- -- -- 0 0 -- E R
v1_jupytergpu 32gb -- 08:00:00 1 10 8 -- E R
v1_interactive 64gb -- 08:00:00 1 0 0 -- E R
v1_ood 64gb -- 08:00:00 1 6 0 -- E R
carbon -- -- -- -- 0 0 -- E R
----- -----
4806 25854
real 3m26.643s
user 0m0.004s
sys 0m0.005s
-------------------------------------------------
$ time qstat -q
server: pbs-7
Queue Memory CPU Time Walltime Node Run Que Lm State
---------------- ------ -------- -------- ---- ----- ----- ---- -----
v1_gpu72 920gb -- 72:00:00 1 75 491 -- E R
v1_small24 128gb -- 24:00:00 1 519 23349 -- E R
v1_small24a 128gb -- 24:00:00 1 877 205 -- E R
v1_small72 128gb -- 72:00:00 1 198 244 -- E R
v1_small72a 128gb -- 72:00:00 1 2565 693 -- E R
v1_medium24 450gb -- 24:00:00 1 26 196 -- E R
v1_medium24a 450gb -- 24:00:00 1 45 9 -- E R
v1_medium72 450gb -- 72:00:00 1 103 34 -- E R
v1_medium72a 450gb -- 72:00:00 1 226 28 -- E R
v1_large24 920gb -- 24:00:00 1 16 101 -- E R
v1_large24a 920gb -- 24:00:00 1 5 55 -- E R
v1_large72 920gb -- 72:00:00 1 53 31 -- E R
v1_large72a 920gb -- 72:00:00 1 65 162 -- E R
v1_largemem72 4000g -- 72:00:00 1 16 22 -- E R
v1_capability24 2048g -- 24:00:00 4 12 16 -- E R
v1_capability48 2048g -- 48:00:00 4 11 27 -- E R
v1_jupyter 64gb -- 08:00:00 1 18 13 -- E R
cx 4000g -- 72:00:00 4 0 1 -- E R
system_queue -- -- -- -- 0 0 -- E R
v1_jupytergpu 32gb -- 08:00:00 1 11 8 -- E R
v1_interactive 64gb -- 08:00:00 1 0 0 -- E R
v1_ood 64gb -- 08:00:00 1 6 0 -- E R
carbon -- -- -- -- 0 0 -- E R
----- -----
4847 25685
real 0m0.456s
user 0m0.241s
sys 0m0.026s
Environment:
Nextflow version: 26.04.6
Operating system: RHEL 8.10
Bash version: GNU bash, version 4.4.20(1)-release (x86_64-redhat-linux-gnu)
Additional context
The suggested fix is to remove any use of the -f flag in any qstat commands that are at a risk of being run as is.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in modules/nextflow/src/main/groovy/nextflow/executor/PbsProExecutor.groovy around line 102 and inspect the polling commands, then read the related cases in modules/nextflow/src/test/groovy/nextflow/executor/PbsProExecutorTest.groovy around line 214. Confirm which qstat invocations use verbose output, update the relevant behavior, and run the executor tests to verify polling no longer requests unnecessary full listings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100