nextflow-io / nextflow-io/nextflow

nextflow 26.04.x+ incompatible with LSF 10.1.0, Stopping a head job fails to stop compute jobs.

Open
#7,640 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

executor/lsf triage/investigate
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

(Please follow this template by replacing the text between parentheses with the requested information)

Expected behavior and actual behavior

Expected: Terminating a workflow head job should also cancel all compute jobs
Actual: Failed to terminate compute jobs with error bkill: Illegal option -- d

Steps to reproduce the problem
  • We are using seqra platform to launch a workflow. Start any workflow normally with LSF platform 10.1.0 compute.
  • Cancel the workflow from seqera UI.
  • Monitor the lsf jobs
Program output
Sept-14 10:49:02.883 [SIGTERM handler] WARN  n.processor.TaskPollingMonitor - Killing running tasks (1)
Sept-14 10:49:02.971 [SIGTERM handler] DEBUG n.executor.AbstractGridExecutor - Unable to kill pending jobs
- cmd executed: bkill -d 5322993
- exit status : 255
- output      :
  bkill: Illegal option -- d
Environment
  • Nextflow version: [26.04.0]
  • Java version: [?]
  • Operating system: [Linux]
  • Bash version: (*)
Additional context

This is a follow up from support ticket.

The issue only happens when we run nextflow version 26.04.3 but works fine on 25.04.2, This is due to our LSF platform is running on version 10.1.0.which does not recognise the option -d . This is stopping us from upgrading nextflow version to 26+ in our production environment.

This is the Pull Request where the issue was introduced

Our lsf version:

lsid
IBM Spectrum LSF Standard 10.1.0.8, May 10 2019

An easier solution might is to introduce a environment variable specific to this issue so it does not add -d to bkill command.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the LSF job-cancellation path and compare the bkill invocation introduced by pull request #6784 with the behavior in Nextflow 25.04.2. Reproduce workflow cancellation against LSF 10.1.0 and verify that stopping the head job also cancels compute jobs without the unsupported -d option.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
hpc
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.