dask / dask/dask-jobqueue

More graceful job cancellation

Open
#640 4 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
256
Forks
150
PR merge metrics
No merged PRs in 30d

Description

Hey all,

This is just a thought for the `SLURMCluster` for now (since that's what I'm familiar with) but similar options may be available in other clusters too. Currently, the `cancel_command` in the `SLURMJob` class is a bare `"scancel"`.

https://github.com/dask/dask-jobqueue/blob/8713202488c664452bf0883bcd4f776536644676/dask_jobqueue/slurm.py#L15

This means that, even when workers are shutdown completely gracefully, the Slurm job is marked as `CANCELLED`. Instead, if the command were `scancel --signal=SIGTERM` the job would be marked as `COMPLETED`. Its possible there could be cases where we would want a job to cancelled, which complicates this somewhat.

In the simple case, however, I think this could be implmented with a simple change of `cancel_command` to:
```python
class SLURMJob(Job):
# Override class variables
submit_command = "sbatch"
cancel_command = "scancel --signal=SIGTERM"
config_name = "slurm"
```

It'd be great to get some more thoughts on the implications for this.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.