dask / dask/dask-jobqueue

LSF broken for runtimes given in HH:MM:SS format

Open
#703 1 comment 0 reactions 0 assignees View on GitHub
bug LSF
Dominant language
Python
Stars
256
Forks
150
PR merge metrics
No merged PRs in 30d

Description

I ran into a really strange issue all of a sudden which took me quite a while to figure out.

Turns out, I had changed from using a `walltime="01:00"` to a walltime of `walltime="01:00:00"`. This is unexpected because SLURM supports the HH:MM:SS format and some of the documentation presents walltimes in this format.

Currently the walltime seems to be passed down directly to the generated jobscript.

## Example
```py
cluster = LSFCluster(
queue="hpc",
cores=4,
memory="8GB",
walltime="01:00:00",
)
client = Client(
cluster,
)
```

Produces:
```text
2026-07-28 16:09:26,984 - dask_jobqueue.lsf - DEBUG - Job script:
#!/usr/bin/env bash

#BSUB -J dask-worker
#BSUB -q hpc
#BSUB -n 4
#BSUB -R "span[hosts=1]"
#BSUB -M 8000
#BSUB -W 01:00:00
```

Indeed putting any string in there just gets directly interpolated into the jobscript (tested with `01:abc:00`).

Unfortunately misconfiguration gives a very misleading error trace:
```

2026-07-28 16:17:22,553 - dask_jobqueue.core - DEBUG - Executing the following command to command line
bsub< /tmp/tmpbydzjudm.sh 2> /dev/null
2026-07-28 16:17:22,565 - tornado.application - ERROR - Exception in callback functools.partial(>, exception=RuntimeError('Command exited with non-zero exit code.\nExit code: 255\nCommand:\nbsub< /tmp/tmpbydzjudm.sh 2> /dev/null\nstdout:\n\nstderr:\n\n')>)
Traceback (most recent call last):
File "/zhome/1a/a/219376/Code/qim-naturarv3d-compression/.pixi/envs/default/lib/python3.14/site-packages/tornado/ioloop.py", line 758, in _run_callback
ret = callback()
File "/zhome/1a/a/219376/Code/qim-naturarv3d-compression/.pixi/envs/default/lib/python3.14/site-packages/tornado/ioloop.py", line 782, in _discard_future_result
future.result()
~~~~~~~~~~~~~^^
File "/zhome/1a/a/219376/Code/qim-naturarv3d-compression/.pixi/envs/default/lib/python3.14/site-packages/distributed/deploy/spec.py", line 396, in _correct_state_internal
await asyncio.gather(*worker_futs)
File "/zhome/1a/a/219376/Code/qim-naturarv3d-compression/.pixi/envs/default/lib/python3.14/asyncio/tasks.py", line 723, in _wrap_awaitable
return await awaitable
^^^^^^^^^^^^^^^
File "/zhome/1a/a/219376/Code/qim-naturarv3d-compression/.pixi/envs/default/lib/python3.14/site-packages/distributed/deploy/spec.py", line 74, in _
await self.start()
File "/zhome/1a/a/219376/Code/qim-naturarv3d-compression/.pixi/envs/default/lib/python3.14/site-packages/dask_jobqueue/core.py", line 426, in start
out = await self._submit_job(fn)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/zhome/1a/a/219376/Code/qim-naturarv3d-compression/.pixi/envs/default/lib/python3.14/site-packages/dask_jobqueue/lsf.py", line 111, in _submit_job
return await self._call(piped_cmd, shell=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/zhome/1a/a/219376/Code/qim-naturarv3d-compression/.pixi/envs/default/lib/python3.14/site-packages/dask_jobqueue/core.py", line 514, in _call
raise RuntimeError(
...<5 lines>...
)
RuntimeError: Command exited with non-zero exit code.
Exit code: 255
Command:
bsub< /tmp/tmpbydzjudm.sh 2> /dev/null
stdout:

stderr:
```

## Proposed solution
I think it would be nice if the runtime as parsed as a `timedelta` and then rendered into either `HH:MM` or just minutes for the LSFCluster specifically. Alternatively, just checking whether the format is HH:MM:SS for `LSFCluster` and raising an error that clearly explains this footgun would be a good solution as well.

If there is appetite and one of the solutions above is preferred, I would be happy to submit a PR for it.

Relevant LSF documentation: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=o-w-1

Contributor guide

Open the contributing guide

Research direction

Start at the LSFCluster entry point and inspect how its walltime value is passed into the generated job script. Compare the accepted runtime formats with the linked LSF documentation and the shown 01:00:00 and invalid-value examples. Done means valid input is handled appropriately and invalid input produces a clear, actionable result.

Written by the indexing model from the issue text.

Assessment

Tech stack
hpc, python
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.