NVIDIA-NeMo / NVIDIA-NeMo/Curator
Resuming the job on slurm after it gets cancelled.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
Initial discussion happened with @VibhuJawa
Is your feature request related to a problem? Please describe.
While running a workflow on slurm with large files, if it needs to be cancelled or stopped due to 4 hr slurm job restriction, we will have partial results. Can we have robust resume feature which can handle even if files are partially processed.
Describe the solution you'd like
Merging output file parts (0.part, 1.part, ...) into 1 file and compare with input file for checking whether complete processing has happened or not, if not then while job resumes the remaining file part will also be counted in set which will go for processing.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating the Slurm workflow execution and output-part handling, then trace how cancellation and restart currently behave; done should include safely combining completed parts, detecting incomplete input processing, and resuming only the remaining parts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100