llnl / llnl/scr

python: Generalize scr watchdog

Open
#402 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

python
Dominant language
C
Stars
108
Forks
37
PR merge metrics
No merged PRs in 30d

Description

The current watchdog implementation is specific to SLURM. It assumes the MPI job was launched with ``srun``, and it queries for the corresponding jobstep id. Then when it detects the job is hanging, it runs ``scancel`` on the jobstep id.

We need to generalize this to support job launchers other than SLURM. I think that most MPI job launchers will tear down an MPI job if one sends a signal like ``SIGINT`` or ``SIGKILL`` to the pid of the job launcher command (``mpirun``, ``jsrun``, etc). We should implement that method as the default and test it on SLURM ``srun``, LSF ``jsrun``, SLURM with MVAPICH2 ``mpirun_rsh``.

As a bonus, we could see whether it's possible to keep the existing SLURM-specific method as a specialization on the JobLauncher class if one used ``srun`` to launch a job.

Let's move the jobstep_id and kill_jobstep methods out of the resource manager class. Those should go in the job luancher class I think.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the JobLauncher and resource manager implementations, focusing on jobstep_id and kill_jobstep. Compare the launcher-pid signal behavior with srun, jsrun, and mpirun_rsh; done means the default watchdog works across those launchers and the SLURM-specific behavior remains available for srun.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.