NVIDIA-NeMo / NVIDIA-NeMo/Curator

UCX segfault during Ray shutdown on SLURM/Apptainer

Open
#2,073 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug community-request waiting-on-customer
Dominant language
Python
Stars
1.8k
Forks
328
Avg merge
4d 5h
Merged PRs (30d)
30

Description

Describe the bug
NeMo Curator pipelines running on SLURM with Apptainer containers complete all stages successfully, but crash with a segmentation fault (signal 11) during Ray cluster shutdown. This causes SLURM to mark the job as FAILED despite all data being correctly written, which breaks afterok dependency chains for post-pipeline jobs (report generation, HF sync).

Steps/Code to reproduce bug
The crash occurs intermittently (~50% of runs, sample size 3) after the pipeline completes and SlurmRayClient calls ray.shutdown(). All output data is valid on disk.

Expected behavior
Pipeline that completes all stages successfully should exit 0.
Instead, Ray shutdown triggers a SIGSEGV in UCX libucs.so, causing SLURM to mark the job as FAILED.

Additional context
The stack trace, related issues (openucx/ucx#7283, ray-project/ray#6025, ray-project/ray#59551), impact on afterok chains, and the questions about UCX_TLS=tcp.

Example of stacktrace:

[<hostname-here>:2186963:0:2290267] Caught signal 11 (Segmentation fault: address not mapped to object at address 0xa0)
==== backtrace (tid:2290267) ====
 0  /opt/venv/lib/python3.12/site-packages/libucx/lib/libucs.so(ucs_handle_error+0x294)
 1  /opt/venv/lib/python3.12/site-packages/libucx/lib/libucs.so(+0x34cca)
 2  /opt/venv/lib/python3.12/site-packages/libucx/lib/libucs.so(+0x34f7e)
 3  /lib/x86_64-linux-gnu/libc.so.6(+0x45330)
 4  /opt/venv/lib/python3.12/site-packages/ray/_raylet.so(+0xcae8cd)
 5  /opt/venv/lib/python3.12/site-packages/ray/_raylet.so(+0xcae997)
 6  /opt/venv/lib/python3.12/site-packages/ray/_raylet.so(_ZN3ray6RayLogD1Ev+0x8f)
 7  /opt/venv/lib/python3.12/site-packages/ray/_raylet.so(+0x10f4ada)
 8  /opt/venv/lib/python3.12/site-packages/ray/_raylet.so(+0x111a6ff)
 9  /opt/venv/lib/python3.12/site-packages/ray/_raylet.so(+0x1122ec6)
10  /opt/venv/lib/python3.12/site-packages/ray/_raylet.so(+0x11197a5)
11  /opt/venv/lib/python3.12/site-packages/ray/_raylet.so(+0x17db5a0)
12  /lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)
13  /lib/x86_64-linux-gnu/libc.so.6(+0x129c3c)
=================================

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the SlurmRayClient shutdown path around ray.shutdown() and compare its behavior with the provided UCX/Ray stack trace and linked issues. Reproduce the intermittent failure across several SLURM/Apptainer runs; done means a pipeline that completes successfully exits 0 so SLURM afterok dependencies continue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.