nextflow-io / nextflow-io/nextflow
nextflow 26.04.x+ incompatible with LSF 10.1.0, Stopping a head job fails to stop compute jobs.
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Bug report
(Please follow this template by replacing the text between parentheses with the requested information)
Expected behavior and actual behavior
Expected: Terminating a workflow head job should also cancel all compute jobs
Actual: Failed to terminate compute jobs with error bkill: Illegal option -- d
Steps to reproduce the problem
- We are using seqra platform to launch a workflow. Start any workflow normally with LSF platform 10.1.0 compute.
- Cancel the workflow from seqera UI.
- Monitor the lsf jobs
Program output
Sept-14 10:49:02.883 [SIGTERM handler] WARN n.processor.TaskPollingMonitor - Killing running tasks (1)
Sept-14 10:49:02.971 [SIGTERM handler] DEBUG n.executor.AbstractGridExecutor - Unable to kill pending jobs
- cmd executed: bkill -d 5322993
- exit status : 255
- output :
bkill: Illegal option -- d
Environment
- Nextflow version: [26.04.0]
- Java version: [?]
- Operating system: [Linux]
- Bash version: (*)
Additional context
This is a follow up from support ticket.
The issue only happens when we run nextflow version 26.04.3 but works fine on 25.04.2, This is due to our LSF platform is running on version 10.1.0.which does not recognise the option -d . This is stopping us from upgrading nextflow version to 26+ in our production environment.
This is the Pull Request where the issue was introduced
Our lsf version:
lsid
IBM Spectrum LSF Standard 10.1.0.8, May 10 2019
An easier solution might is to introduce a environment variable specific to this issue so it does not add -d to bkill command.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the LSF job-cancellation path and compare the bkill invocation introduced by pull request #6784 with the behavior in Nextflow 25.04.2. Reproduce workflow cancellation against LSF 10.1.0 and verify that stopping the head job also cancels compute jobs without the unsupported -d option.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100