NVIDIA-NeMo / NVIDIA-NeMo/Run

Allow RayJobs to connect to existing RayClusters with different names

Open
#362 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
258
Forks
113
Avg merge
1d 3h
Merged PRs (30d)
5

Description

Is your feature request related to a problem? Please describe.
The documentation for launching RayJobs shows the steps to first deploy a RayCluster with a given name (demo-slurm-ray). This spins up a RayCluster using the executor of choice. It then creates a RayJob with a new name (ie. demo-slurm-job). The RayJob code checks if there is a RayCluster with the same name as the RayJob and will run it in the existing cluster if so. Otherwise, if the cluster with the same name as the job doesn't exist, it will create a new ephemeral cluster specifically for that job.

There doesn't appear to be a way in the code to link a RayJob with an existing cluster, so users are forced to use the same name for their RayJob as the name of their RayCluster which isn't ideal as all jobs would have the same name.

Describe the solution you'd like
There should be a way to specify the existing RayCluster name if it doesn't already exist. For example, maybe there is a new cluster_name parameter to the RayJob base class which allows users to specify which cluster to try and connect to if it exists.

Describe alternatives you've considered
The RayJob could have another dependency on the RayCluster class so submitting jobs is one of the methods from within RayCluster, but that doesn't feel like a good solution.

Additional context
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the RayJob base class behavior that looks up a RayCluster by the RayJob name, then review the RayCluster class relationship and the RayJobs documentation in docs/source/guides/ray.md. Define how an explicit existing cluster name should be selected while preserving ephemeral-cluster behavior, and verify the documented Slurm quick-start flow with separate RayJob and RayCluster names.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.