NVIDIA-NeMo / NVIDIA-NeMo/RL

Add PBS Scheduler Support

Open
#992 3 comments 0 reactions 1 assignee Claimed by @terrykong View on GitHub
external r0.6.0 x-nla x-oko
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Is your feature request related to a problem? Please describe.**

While NeMo-RL provides excellent [Slurm documentation](https://docs.nvidia.com/nemo/rl/latest/cluster_start.html) and examples, there's no equivalent support for PBS environments. The main challenge is Nemo RL's dependency on Ray, which is difficult to configure and use effectively in PBS environments, even when it's technically possible to get it working. This creates a significant barrier for researchers who want to use it on academic clusters.

**Describe the solution you'd like**

I would like PBS-specific documentation and examples similar to what exists for Slurm or an even better approach would be a path that doesn't depend on Ray.

**Describe alternatives you've considered**

- **Manual Ray setup** - Tried configuring Ray manually on PBS but ran into issues with node discovery and resource allocation
- **In-framework execution path** - An alternative solution could be developing a Megatron based execution path that doesn't rely on Ray

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.