NVIDIA-NeMo / NVIDIA-NeMo/Curator
Add features from Llama Nemotron tutorial to NeMo Curator modules
@sarahyurick is already working on this.
Since May 12, 2025.
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
While some of the classes in this tutorial are very specific to the dataset being curated, others could be useful for a variety of datasets and should be added as NeMo Curator modules.
In particular, I think it would be nice to add these as modules: NonEnglishFilter (could be generalized to any language), TokenCountFilter (Curator already has a filter called TokenCountFilter, so the one from the tutorial would need to be renamed/consolidated with the existing one), and CompletionTokenCountFilter.
Additionally, we should try to take advantage of batched processing with AutoTokenizer (including for the existing TokenCountFilter filter).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.