NVIDIA-NeMo / NVIDIA-NeMo/RL

Ray dashboards won't load due to symlinked files during install

Open
#1,476 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Describe the bug**

Trying to open the Ray dashboard in the NeMo-RL container will result in 404 errors and the dashboard will never load. This is due to Ray being installed in the `/opt/nemo_rl_venv` directory by `uv` but the main.js and main.css files used for the Ray dashboard get symlinked to `/root` as the container install does symlinks for all copies in the `uv` stages in the Dockerfile. The HTTP server used by Ray dashboard doesn't allow symlinks outside of the parent directory which causes the 404 errors. Instead, by using the `copy` link-mode during the `uv pip install` for Ray will fix the error.

**Steps/Code to reproduce bug**

1. Build the `nemo-rl` release container locally. Tested with commit `5514d1e6eb0036f4ccbc5dac42b1b860959d0087`.
2. Launch the container locally and expose port `8265` for the Ray dashboard.
3. Run `ray start --head --port=6379 --dashboard-host=0.0.0.0 --block --include-dashboard=true` inside the container to start the Ray cluster.
4. Open localhost:8265 in your browser to view the Ray dashboard - this will never load and there will be 404 errors in the browser debug tools.

Instead, this can be worked around by re-installing the `ray` package with `uv` and using `copy` link-mode. To do so:

1. Stop the Ray process in the container if still running.
2. Re-install ray with `uv pip install --link-mode=copy --force-reinstall "ray[default]==2.46.0"`.
3. Re-launch the Ray cluster with `ray start --head --port=6379 --dashboard-host=0.0.0.0 --block --include-dashboard=true`.
4. Open localhost:8265 in your browser - the dashboard should now load properly.

**Expected behavior**

Ray should be installed without symlinks so it can render the dashboard properly. As-is, the dashboard will not load due to the symlinks being in another parent directory.

The Dockerfile should be updated to either force-reinstall Ray without symlinks, or all dependences should be installed without symlinks, but that is less desirable.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.