Intermittent segfault in libcuda.so.1 during PPO training with custom EnvBase + ParallelEnv workers

Training crashes with a segmentation fault at unpredictable step counts (observed between 25 and ~1019 steps) during PPO training using a custom EnvBase subclass with num_workers parallel environments. The crash occurs inside the NVIDIA CUDA driver itself (libcuda.so.1), triggered by an ordinary tensor multiplication during the policy/value forward pass — not during my environment's own code.

Environment:

• torch: 2.5.1+cu121 (also reproduced identically on 2.7.1+cu128)
• torchrl: 0.13.3
• NVIDIA driver: 535.309.01
• GPU: NVIDIA RTX 6000 Ada Generation
• CUDA (driver-supported): 12.2
• OS: Ubuntu 24.04
• Python: 3.11
• num_workers: reproduces with both 1 and 4

What I've ruled out:

• A debug print statement calling repr() on a CUDA tensor every step (removed, crash persisted)
• OpenMP/MKL/OpenBLAS thread oversubscription (OMP_NUM_THREADS=1 etc., no effect)
• Heap corruption via glibc malloc (MALLOC_CHECK_=3, no abort triggered, same crash)
• CUDA/driver version mismatch (downgraded torch to match driver's max supported CUDA 12.2, crash persisted, even at a lower step count)
• Number of parallel workers (crashes identically with num_workers=1 and num_workers=4)
• Disabling the evaluation rollout entirely (crash still reproduces without eval ever running)
• A custom free_memory() utility calling torch.cuda.ipc_collect() — confirmed not even in the executed code path
• Classic GPU memory violations — ran under compute-sanitizer --tool memcheck, which reported 0 errors, yet the process still segfaulted

Backtrace (gdb) — crash occurs inside libcuda.so.1 with no exported symbol:

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffea842c680 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1

Frames above it show the call chain through PyTorch's autograd dispatcher:

#13 c10::impl::wrap_kernel_functor_unboxed_<...mul_Tensor...>::call(...)
#14 at::_ops::mul_Tensor::call(at::Tensor const&, at::Tensor const&) ()
#15 torch::autograd::THPVariable_mul(...)

Python-level faulthandler traceback (representative example):

Current thread (most recent call first):
File ".../src/envs/environment.py", line 310 in _step
File ".../torchrl/envs/transforms/_base.py", line 1349 in _step
File ".../torchrl/envs/common.py", line 2353 in step
File ".../torchrl/envs/common.py", line 4107 in _step_and_maybe_reset
File ".../torchrl/envs/common.py", line 4061 in step_and_maybe_reset
File ".../torchrl/collectors/_single.py", line 1814 in rollout
File ".../torchrl/_utils.py", line 652 in unpack_rref_and_invoke_function
File ".../src/agents/ppo/train.py", line 256 in train

Reproduction pattern:

• Uses SyncDataCollector with a custom EnvBase subclass wrapping a native simulation engine (ray-tracing/geometry-based, via trimesh)
• Crash frequency/timing is not deterministic — same code, same config, different step counts each run (25, 38, 71, 123, 157, 196, 1019 observed)
• Also see a consistent resource_tracker: 4 leaked semaphore objects warning at shutdown, independent of num_workers count