futureverse / futureverse/future.batchtools
Slurm template: Signal SIGINT by default to allow R code to catch it gracefully
- Lingua principale
- R
- Stelle
- 87
- Fork
- 10
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
When a job is approaching it's maximum run-time limit, Slurm sends a `SIGTERM` 30 seconds before sending a `SIGKILL` (abrupt; not capturable).
R code cannot handle `SIGTERM` signals, only `SIGINT`. If Slurm would signal `SIGINT` instead, we could capture it internally as an `interrupt` condition, and using `tryCatch(..., interrupt = ...)`, `on.exit()`, and likes to gracefully exit, e.g. close connections, checkpoint intermediate results, etc.
Slurm allows us to declare what type of signal, and when, to signal when we approach the run-time limit. This can be done by declaring, e.g. `--signal=B:INT@60`.
# Idea
First, should be able to control the signal explicitly via:
```r
plan(..., resources = list(signal = "INT@60"))
```
already today.
Second, we could update the default to be `signal = "INT@60"` by adding the following to the template:
```bash
## Resources needed
<%
## Default to sending SIGINT 60 seconds before walltime limit
## to allow graceful R-level cleanup/checkpointing
if (is.null(resources[["signal"]])) {
resources[["signal"]] <- "B:INT@60"
}
...
%>
```
Third, alternative to a Slurm-specific resource name, we might harmonize the signal type and signal grace period with what is used by other job schedulers.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Locate the Slurm template and the resource handling behind plan(..., resources = list(signal = ...)); start by checking how scheduler-specific defaults and explicit signal values are currently passed through. Done means the Slurm default uses B:INT@60 while an explicitly supplied signal remains effective, with relevant existing tests updated or added if the project has coverage for templates.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- r
- Ambito
- distributed-systems, infrastructure
- Tipo di issue
- Funzionalità
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 66/100