NVIDIA-NeMo / NVIDIA-NeMo/Automodel
Make StatefulDataloader be DP-aware
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
Is your feature request related to a problem? Please describe.
StatefulDataloader is currently not DP-aware. If you change DP size during a checkpoint then you will get inconsistent results because StatefulDataloader only tracks its own (local to DP group) samples. We need to extend this to enable global dataloader resharding.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating StatefulDataloader and tracing how it records samples during checkpointing within a data-parallel group. Define and validate global resharding behavior for checkpoints restored with a different DP size, including consistent sample results; the issue does not name specific files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100