python: Generalize scr watchdog
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 108
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
The current watchdog implementation is specific to SLURM. It assumes the MPI job was launched with ``srun``, and it queries for the corresponding jobstep id. Then when it detects the job is hanging, it runs ``scancel`` on the jobstep id.
We need to generalize this to support job launchers other than SLURM. I think that most MPI job launchers will tear down an MPI job if one sends a signal like ``SIGINT`` or ``SIGKILL`` to the pid of the job launcher command (``mpirun``, ``jsrun``, etc). We should implement that method as the default and test it on SLURM ``srun``, LSF ``jsrun``, SLURM with MVAPICH2 ``mpirun_rsh``.
As a bonus, we could see whether it's possible to keep the existing SLURM-specific method as a specialization on the JobLauncher class if one used ``srun`` to launch a job.
Let's move the jobstep_id and kill_jobstep methods out of the resource manager class. Those should go in the job luancher class I think.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the JobLauncher and resource manager implementations, focusing on jobstep_id and kill_jobstep. Compare the launcher-pid signal behavior with srun, jsrun, and mpirun_rsh; done means the default watchdog works across those launchers and the SLURM-specific behavior remains available for srun.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, linux
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100