NVIDIA-NeMo / NVIDIA-NeMo/Curator
Supporting Anyscale to run NeMo Curator Jobs
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
Currently NeMo Curator can be used on Anyscale, however there is some work that needs to be done to support it better
Gotchya
- Set num_cpus / num_gpus on head node to zero
- Set higher idle node timeout
- Ensure no stage has
num_cpus <= 0otherwise task can be scheduled on head node (See https://github.com/NVIDIA-NeMo/Curator/pull/1209#issuecomment-3483014428)
TODO
- Allow executors to not schedule on head node / ignore head node (see anyscale docs)
- Ray Data and Ray Actor Pool https://github.com/NVIDIA-NeMo/Curator/pull/1209
- Xenna
- Improve Cloud I/O Support
- [ ] #1214
- [ ] #1217
- [ ] #1213
- [ ] Look into if Anyscale supports mounting cloud paths s.t. all paths are "local" - See if notebook vs submitting Ray Jobs are any different?
- Have some form of QA
- Have dedicated documentation for Anyscale
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Review the TODO list, starting with Ray Data and Ray Actor Pool support in PR #1209, then inspect cloud I/O issues #1214, #1217, and #1213; the work is done when Anyscale execution, QA, and dedicated documentation are addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud, data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100