NVIDIA-NeMo / NVIDIA-NeMo/Curator
Optimize `merge_file_prefixes` for merging bin/idx files
Open
@sarahyurick is already working on this.
Since May 13, 2026.
enhancement
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 327
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
While testing https://github.com/NVIDIA-NeMo/Curator/pull/1727, I noticed that https://github.com/NVIDIA-NeMo/Curator/blob/main/nemo_curator/utils/merge_file_prefixes.py is a large bottleneck. From some light investigations, it seems like there are a few places where we can add parallelism and improve the runtime.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.