vLLM worker crashes (SIGSEGV) during initialization on multi-node Ray clusters (8+ nodes)

Summary

When running the Nemotron-Parse PDF pipeline with vLLM inference across 8 or more nodes on a Ray cluster (via Cosmos-Xenna), one or more vLLM worker actors crash with SIGSEGV during engine initialization. The crash is not an OOM — nodes have 2 TB RAM with only ~5.5% used at crash time. The crash is probabilistic: more nodes means higher probability of at least one failure. 4-node runs succeed consistently.

This appears related to the torch.compile race condition addressed in PR #1590, but the fix applied there (serializing initialization via setup_on_node) does not fully resolve the issue.

Environment

• Container: nvcr.io/nvidia/nemo-curator:26.02
• Hardware: DGX H100 nodes (8× H100 80GB GPUs, 128 CPUs, 2 TB RAM each)
• vLLM: v0.17.0 (V1 engine)
• Model: nvidia/NVIDIA-Nemotron-Parse-v1.2 (Qwen2.5-VL based, bf16)
• vLLM config: tensor_parallel_size=1, compilation_mode=VLLM_COMPILE, enforce_eager=False
• Ray: Cluster formed via SlurmRayClient on Slurm
• Xenna: Cosmos-Xenna executor (streaming mode)

Pipeline Architecture

The pipeline has 5 stages, with the GPU inference stage being the one that crashes:

1. PDFPartitioningStage — CPU, reads manifest, creates file group tasks
2. PDFPreprocessStage — CPU, extracts PDFs from zips, renders pages to images
3. NemotronParseInferenceStage — GPU (1 GPU per worker, 8 workers per node), runs vLLM
4. NemotronParsePostprocessStage — CPU, formats model outputs
5. InterleavedParquetWriterStage — CPU, writes output parquet files

Xenna allocates one inference worker per GPU (8 per node). Each worker initializes its own vLLM LLM engine with tensor_parallel_size=1.

Reproduction

What works: 4 nodes (32 GPUs)

• Job 9927635: 83,390 PDFs on 4 nodes — completed successfully in 2h 10min (no setup_on_node fix)
• Job 10377659: 1,000,000 PDFs on 4 nodes — completed successfully (with setup_on_node fix)

What fails: 8+ nodes

Job ID	Nodes	setup_on_node fix?	Result	Time to crash
10358110	40	No	CRASHED	~30s after init start
10358111	40	No	CRASHED	~30s after init start
10358112	40	No	CRASHED	~30s after init start
10358113	40	No	CRASHED	~30s after init start
10359302	32	No	CRASHED	~30s after init start
10378062	8	Yes	CRASHED	~33s after init start

100% failure rate at 8+ nodes across multiple attempts, with or without the setup_on_node serialization fix.

Error Details

The Ray worker dies with a SYSTEM_ERROR during the vLLM engine initialization phase:

A worker died or was killed while executing a task by an unexpected system error.
Worker IP address: 10.65.16.209
Worker port: 10008
Worker PID: 3707336
Worker exit type: SYSTEM_ERROR
Worker exit detail: Worker unexpectedly exits with a connection error code 2.
End of file. Some common causes include:
  (1) the process was killed by the OOM killer due to high memory usage,
  (2) ray stop --force was called, or
  (3) the worker crashed unexpectedly due to SIGSEGV or another unexpected error.

Key observations

1. Crash always occurs during setup_on_node / vLLM engine initialization, before any data processing starts
2. Crash happens within ~33 seconds of the first setup_on_node call starting
3. Only one node crashes per job — the crash on a single node causes the entire pipeline to abort
4. The crashed node is random — not always the head node:

Job ID	Head Node IP	Crashed Node IP	Same?
10358110	10.65.12.131	10.65.12.211	No
10358111	10.65.3.131	10.65.31.177	No
10358112	10.65.2.211	10.65.2.211	Yes
10358113	10.65.7.45	10.65.2.153	No
10378062	10.65.0.79	10.65.16.209	No

Why this is NOT an OOM

Per-node resource monitoring (30s intervals) was added to the 8-node diagnostic job (10378062). On the crashed node (10.65.16.209) at the time of crash:

              total        used        free      shared  buff/cache   available
Mem:           2007         110        1734           7         161        1875
Swap:             8           0           8

• Total RAM: 2,007 GB (~2 TB)
• Used RAM: 110 GB (5.5%)
• Available RAM: 1,875 GB

GPU memory was also nearly empty (0 MB on 7/8 GPUs, 2.6 GB on 1/8 GPUs) — the vLLM engine hadn't finished loading the model yet.

Timeline of crash (Job 10378062, 8 nodes)

10:29:17.120  Node 10.65.13.5   — setup_on_node: initializing model
10:29:17.698  Node 10.65.27.93  — setup_on_node: initializing model
10:29:18.522  Node 10.65.31.201 — setup_on_node: initializing model
10:29:19.410  Node 10.65.16.143 — setup_on_node: initializing model
10:29:19.442  Node 10.65.16.209 — setup_on_node: initializing model  ← THIS NODE CRASHES
10:29:19.511  Node (head)       — setup_on_node: initializing model
10:29:19.565  Node 10.65.24.221 — setup_on_node: initializing model
10:29:19.802  Node 10.65.16.7   — setup_on_node: initializing model

10:29:41-42   All nodes begin "Initializing a V1 LLM engine (v0.17.0)"
10:29:50-52   Nodes reach "Encoder cache will be initialized" (gpu_model_runner.py:5254)
              Node 10.65.16.209 also reaches this point normally

10:29:52.638  *** CRASH *** Worker PID 3707336 on 10.65.16.209 exits with SYSTEM_ERROR
              "connection error code 2. End of file." (= SIGSEGV)

The crashed worker (PID 3707336) was the PDFPreprocessStage worker on the same node — but the crash is triggered by the vLLM initialization happening on the same node's inference worker (PID 3707343). Note the crashed PID (3707336) differs from the inference worker PID (3707343), suggesting a collateral crash from a shared-memory or CUDA context issue.

Attempted fixes

1. setup_on_node serialization (from PR #1590)

Moved vLLM initialization from setup() (runs per-worker in parallel) to setup_on_node() (runs once per node, serially) with an idempotency guard in setup():

def setup_on_node(self, node_info=None, worker_metadata=None):
    self._initialize_model()

def setup(self, worker_metadata=None):
    if hasattr(self, "_llm") or hasattr(self, "_model"):
        return  # Already initialized by setup_on_node
    self._initialize_model()

Result: Still crashes at 8 nodes. The crash happens during the first (and only) setup_on_node call on a node, so serializing multiple workers doesn't help — the issue is simultaneous initialization across nodes.

2. Xenna resilience settings

Configured XennaExecutor with retry/resilience parameters:

executor = XennaExecutor(config={
    "ignore_failures": True,
    "failures_return_nones": True,
    "reset_workers_on_failure": True,
})

And StageSpec defaults:

StageSpec(
    num_setup_attempts_python=3,
    num_run_attempts_python=3,
    reset_workers_on_failure=True,
    max_setup_failure_percentage=0.20,
)

Plus explicit ActorDiedError catch in the executor to allow graceful exit.

Result: Helps the pipeline not crash hard — but the underlying SIGSEGV still happens, and at 8+ nodes, the cascade of retries and failures still leads to overall pipeline failure.

Hypothesis

The crash is a probabilistic SIGSEGV during vLLM/torch.compile/CUDA initialization. With N nodes, there are N simultaneous vLLM engine initializations (one per node in setup_on_node). Each initialization involves:

1. Loading model weights to GPU
2. torch.compile / Triton kernel compilation
3. CUDA graph capture
4. Encoder cache profiling

The probability of at least one crash appears to scale with the number of concurrent initializations:

• 4 nodes (4 concurrent inits): 0% failure rate (multiple successful runs)
• 8 nodes (8 concurrent inits): 100% failure rate (1 attempt)
• 32 nodes (32 concurrent inits): 100% failure rate (1 attempt)
• 40 nodes (40 concurrent inits): 100% failure rate (4 attempts)

This suggests either:

1. A torch.compile race condition involving shared filesystem locks or cache directories (all nodes share the same Lustre filesystem and HuggingFace cache)
2. A vLLM V1 engine bug triggered probabilistically during CUDA initialization
3. A CUDA driver/runtime issue when many processes simultaneously initialize on the same cluster

Current workaround

Using 4-node jobs with smaller data chunks (~100K PDFs per job) and the --resume flag for fault tolerance. This avoids the crash but significantly limits throughput scalability.

Suggested investigation

1. Try enforce_eager=True to skip torch.compile entirely and see if the crash disappears
2. Try staggering setup_on_node across nodes with a sleep/barrier so only 1-2 nodes initialize at a time
3. Check if separate TORCH_COMPILE_CACHE directories per node prevent the crash
4. Test with vLLM V0 engine (non-V1) to see if the new engine's compilation is the issue
5. Try CUDA_LAUNCH_BLOCKING=1 to surface the real CUDA error behind the SIGSEGV

Related

• PR #1590 — Similar torch.compile race condition fix for video captioning stages

Start by reproducing the multi-node failure around setup_on_node and vLLM engine initialization, comparing 4-node and 8+ node runs. Use the suggested enforce_eager, staggered initialization, separate cache directories, V0 engine, and CUDA_LAUNCH_BLOCKING experiments, and inspect the encoder-cache stage noted at gpu_model_runner.py:5254. Done means identifying a reproducible cause and validating a mitigation beyond the existing PR #1590 serialization fix.