futureverse / futureverse/future.batchtools
Slurm template: Signal SIGINT by default to allow R code to catch it gracefully
- Dominant language
- R
- Stars
- 87
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
When a job is approaching it's maximum run-time limit, Slurm sends a `SIGTERM` 30 seconds before sending a `SIGKILL` (abrupt; not capturable).
R code cannot handle `SIGTERM` signals, only `SIGINT`. If Slurm would signal `SIGINT` instead, we could capture it internally as an `interrupt` condition, and using `tryCatch(..., interrupt = ...)`, `on.exit()`, and likes to gracefully exit, e.g. close connections, checkpoint intermediate results, etc.
Slurm allows us to declare what type of signal, and when, to signal when we approach the run-time limit. This can be done by declaring, e.g. `--signal=B:INT@60`.
# Idea
First, should be able to control the signal explicitly via:
```r
plan(..., resources = list(signal = "INT@60"))
```
already today.
Second, we could update the default to be `signal = "INT@60"` by adding the following to the template:
```bash
## Resources needed
<%
## Default to sending SIGINT 60 seconds before walltime limit
## to allow graceful R-level cleanup/checkpointing
if (is.null(resources[["signal"]])) {
resources[["signal"]] <- "B:INT@60"
}
...
%>
```
Third, alternative to a Slurm-specific resource name, we might harmonize the signal type and signal grace period with what is used by other job schedulers.
Contributor guide
Research direction
Locate the Slurm template and the resource handling behind plan(..., resources = list(signal = ...)); start by checking how scheduler-specific defaults and explicit signal values are currently passed through. Done means the Slurm default uses B:INT@60 while an explicitly supplied signal remains effective, with relevant existing tests updated or added if the project has coverage for templates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 66/100