futureverse / futureverse/future.batchtools

Slurm template: Signal SIGINT by default to allow R code to catch it gracefully

Aperta
#103 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
feature/resources scheduler/slurm
Lingua principale
R
Stelle
87
Fork
10
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

When a job is approaching it's maximum run-time limit, Slurm sends a `SIGTERM` 30 seconds before sending a `SIGKILL` (abrupt; not capturable).

R code cannot handle `SIGTERM` signals, only `SIGINT`. If Slurm would signal `SIGINT` instead, we could capture it internally as an `interrupt` condition, and using `tryCatch(..., interrupt = ...)`, `on.exit()`, and likes to gracefully exit, e.g. close connections, checkpoint intermediate results, etc.

Slurm allows us to declare what type of signal, and when, to signal when we approach the run-time limit. This can be done by declaring, e.g. `--signal=B:INT@60`.

# Idea

First, should be able to control the signal explicitly via:

```r
plan(..., resources = list(signal = "INT@60"))
```

already today.

Second, we could update the default to be `signal = "INT@60"` by adding the following to the template:

```bash
## Resources needed
<%
## Default to sending SIGINT 60 seconds before walltime limit
## to allow graceful R-level cleanup/checkpointing
if (is.null(resources[["signal"]])) {
resources[["signal"]] <- "B:INT@60"
}
...
%>
```

Third, alternative to a Slurm-specific resource name, we might harmonize the signal type and signal grace period with what is used by other job schedulers.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Locate the Slurm template and the resource handling behind plan(..., resources = list(signal = ...)); start by checking how scheduler-specific defaults and explicit signal values are currently passed through. Done means the Slurm default uses B:INT@60 while an explicitly supplied signal remains effective, with relevant existing tests updated or added if the project has coverage for templates.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
r
Ambito
distributed-systems, infrastructure
Tipo di issue
Funzionalità
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
66/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.