NVIDIA-NeMo / NVIDIA-NeMo/Curator
[Ray] GPU CI enhancements for classifier tests
@sarahyurick is already working on this.
Since Aug 21, 2025.
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
In https://github.com/NVIDIA-NeMo/Curator/pull/926, I added safeguards to setup_on_node which was experiencing flakiness when trying to download Hugging Face models.
In the future, we should consider caching the models onto our GPU CI data mounts. For the Aegis and Instruction Data Guard tests, this may be an easy way to mitigate the need for a Hugging Face token.
However, if we do decide to go this route, we still need to ensure that setup_on_node works as expected. I like the idea of having some type of check if the Hugging Face cache exists, else try setup_on_node (this way, a user who clones our repo should still be able to run all tests themselves).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.