[torch_memory_saver.cpp] CUresult error: 2 (out of memory) file=csrc/utils.h func=cu_mem_create line=187

CUDA Out-of-Memory Error During Qwen30B-A3B Training
Environment:

Hardware: 2 machines, 16× H20 GPUs (96GB VRAM each)
Model: qwen30B-A3B
Code Commit: 9a291fca719e6c3fae1be93088b4fa79b774491a (HEAD -> main, origin/main, origin/HEAD)
Date: Tue Sep 2 17:54:39 2025 +0800

Error Context: Training via ray.get(actor_model.async_train(...))
Error Log:[torch_memory_saver.cpp] CUresult error: 2 (out of memory) file=csrc/utils.h func=cu_mem_create line=187

Description:
When training the qwen30B-A3B model with various parallelism configurations (ep, pp, tp, cp), the CUDA OOM error consistently occurs during the async_train call. The error points to memory allocation failure in torch_memory_saver.cpp, despite using high-memory GPUs (H20 96GB).

Is this a sglang-specific issue (e.g., memory fragmentation, Ray actor overhead) or inherent to the model’s memory demands when actor_model.async_train?

Start with the reported allocation failure in csrc/utils.h at line 187 and the surrounding torch_memory_saver.cpp path. Trace the actor_model.async_train entry point under the listed ep, pp, tp, and cp configurations, using the provided environment and commit as the reproduction target. Done means identifying whether the failure is caused by framework memory handling or the model's memory demands, with a documented reproduction and conclusion.