When cancelling a study, if an adapter returns that the job is finishing the status is reflected as running.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 161
- Forks
- 49
- PR merge metrics
- No merged PRs in 30d
Description
While running some tests on an MPI enabled LULESH, I encountered a case where when cancelling a study resulted in inaccurate status reported by Maestro. I don't expect that this case is common, but I wanted to document the states here and the reasons why this happens.
I encountered this using the SLURM adapter, but after looking through the code I realized I made an implicit assumption in the `ExecutionGraph` method `execute_ready_steps`. SLURM has an intermediate status when a job is in the process of wrapping up successfully called "Completing" (which maps to Maestro's enum "FINISHING"). Cancellation works by placing a cancel lock in the study workspace to be caught the next time the conductor wakes up to check status.
There's a corner case where, before the conductor spins up to see the lock AND the jobs in the queue hit the FINISHING state. The `execute_ready_steps` method doesn't have a catch for the FINISHING, which falls through and leaves the steps as RUNNING. The method has an *explicit* check for SLURM's cancelled status and upon seeing it will make the jobs as CANCELLED (Maestro's enumeration). Because the conductor then terminates because the study is cancelled, that RUNNING status that previously fell through is still reported meaning it never gets marked as finished.
A simple solution (but potentially inaccurate) solution to this corner case is to mark a step FINISHED for both a reported FINISHED and FINISHING status. I could see a case where a node could fail in job wrap up and the FINISHED status would then (possibly) be inaccurate. Another way to handle this would be to split apart the status checking that updates the available sets and break down the `execute_ready_steps` method to smaller chunks. This solution is probably the correct one. However, there are plans to update the split of responsibility and move towards multiple DAGs and other tracking structures which could then correct this corner case by nature of just an outright refactor.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the ExecutionGraph.execute_ready_steps method and trace how the SLURM adapter's FINISHING status is handled while the conductor processes cancellation locks. Compare the existing FINISHED and cancelled-status paths, then determine how the corner case should update step status without reporting a failed wrap-up as finished.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100