Help me😭 CUDA error: an illegal memory access was encountered

```
Traceback (most recent call last):
File "grpo_DrugNER2RE.py", line 703, in
main(script_args, training_args, model_args)
File "grpo_DrugNER2RE.py", line 656, in main
train_result = trainer.train(resume_from_checkpoint=checkpoint)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/transformers/trainer.py", line 2241, in train
return inner_training_loop(
^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/transformers/trainer.py", line 3692, in training_step
inputs = self._prepare_inputs(inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/trl/trainer/grpo_trainer.py", line 545, in _prepare_inputs
outputs = self.llm.generate(all_prompts_text, sampling_params=self.sampling_params, use_tqdm=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/utils.py", line 1086, in inner
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 469, in generate
outputs = self._run_engine(use_tqdm=use_tqdm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/entrypoints/llm.py", line 1390, in _run_engine
step_outputs = self.llm_engine.step()
^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/engine/llm_engine.py", line 1386, in step
outputs = self.model_executor.execute_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/executor/executor_base.py", line 138, in execute_model
output = self.collective_rpc("execute_model",
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 51, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/utils.py", line 2220, in run_method
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/worker/worker_base.py", line 413, in execute_model
output = self.model_runner.execute_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:[E403 23:14:34.500719445 ProcessGroupNCCL.cpp:1595] [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f86980b9446 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f86980636e4 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f86981a5a18 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f864e014726 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f864e0193f0 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f864e020b5a in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f864e02261d in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7f869856d5c0 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7f8748888ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7f874891a850 in /lib/x86_64-linux-gnu/libc.so.6)

File "anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
terminate called after throwing an instance of 'c10::DistBackendError'
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1775, in execute_model
output: SamplerOutput = self.model.sample(
^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/model_executor/models/qwen2.py", line 505, in sample
next_tokens = self.sampler(logits, sampling_metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/model_executor/layers/sampler.py", line 288, in forward
maybe_deferred_sample_results, maybe_sampled_tokens_tensor = _sample(
^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/model_executor/layers/sampler.py", line 853, in _sample
return _sample_with_torch(
^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/model_executor/layers/sampler.py", line 822, in _sample_with_torch
return get_pythonized_sample_results(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/model_executor/layers/sampler.py", line 687, in get_pythonized_sample_results
sample_results = _random_sample(seq_groups,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "anaconda3/envs/open-r1/lib/python3.11/site-packages/vllm/model_executor/layers/sampler.py", line 486, in _random_sample
random_samples = random_samples.cpu()
^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
what(): [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f86980b9446 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f86980636e4 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f86981a5a18 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7f864e014726 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7f864e0193f0 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7f864e020b5a in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f864e02261d in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0x145c0 (0x7f869856d5c0 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame #8: + 0x94ac3 (0x7f8748888ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #9: + 0x126850 (0x7f874891a850 in /lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f86980b9446 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: + 0xe4271b (0x7f864dc8f71b in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7f869856d5c0 in anaconda3/envs/open-r1/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame #3: + 0x94ac3 (0x7f8748888ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: + 0x126850 (0x7f874891a850 in /lib/x86_64-linux-gnu/libc.so.6)
```

When I train to the second or third epoch, an error suddenly occurs and the training stops. My training environment uses an L40 GPU with 47GB of memory, and during training, about 42GB of memory is used. My vLLM version is 0.7.2. I have successfully completed training before, but this time the problem suddenly appeared and I can’t figure out how to fix it. I’d really appreciate any help.

Start by reproducing the failure from grpo_DrugNER2RE.py around lines 656 and 703, using CUDA_LAUNCH_BLOCKING=1 as suggested in the traceback. Inspect the vLLM path through worker/model_runner.py and model_executor/layers/sampler.py, especially _random_sample where random_samples.cpu() fails. Done means the triggering condition is identified and a verified training run completes past the second or third epoch.

Written by the indexing model from the issue text.