Some "TIMED OUT" jobs not restarting properly
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 161
- Forks
- 49
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I'm having a weird bug happen when Maestro tries to restart jobs.
Say, I've launched one study with 8 jobs in the study. When these 8 jobs timeout, maybe half of them restart successfully (meaning they are resubmitted to Slurm, and the "TIMED OUT" status changes back to "RUNNING" and the number of restarts goes up).
The other half of the jobs never register as "TIMED OUT",and are never resubmitted back to Slurm. The `maestro status` command still shows them as "RUNNING", but does not increment the number of restarts. The jobs also no longer show up in the `study.log` file.
Something to note is that the initial runs of these jobs typically all end within a few minutes of each other.
Hopefully I've provided enough information here to help figure this out.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported case with a study containing 8 jobs, then trace Maestro's timeout handling and Slurm resubmission path alongside the `maestro status` output and `study.log`. Done means every timed-out job is recorded as `TIMED OUT`, resubmitted to Slurm, returned to `RUNNING`, and counted as restarted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100