set `PYTORCH_CUDA_ALLOC_CONF` for 80GB GPUs due to logits memory usage

Hi, we recently found a useful trick when running the current version of slime on 80GB GPUs.

When using slime, the framework saves a seq_len * vocab_size logits tensor. However, because torch_memory_saver disables expandable_segments, this can lead to CUDA OOM issues.

Before running, it is necessary to estimate the required max_split_size_mb. Without proper tuning, the job may fail due to insufficient contiguous memory blocks.

A reasonable estimate of max_split_size_mb can be obtained with the formula:

max_split_size_mb ≈ (seq_len / CP) * vocab_size * 2 bytes

• seq_len = response sequence length
• CP = context parallelism size
• vocab_size = vocabulary size

Example (RL with Qwen3)

• max response length seq_len = 16384
• training CP = 4
• Qwen3 vocab_size = 151936

Then:

16384 / 4 * 151936 ≈ 1 GB

So, for 80GB GPUs, it is recommended to add this environment variable in ray submit to avoid fragmentation and OOM errors:

"PYTORCH_CUDA_ALLOC_CONF": "max_split_size_mb:1024"

Root Cause(I guess...)

If max_split_size_mb is smaller than the logits tensor size, then allocating the logits tensor requires multiple memory blocks. Since PyTorch’s CUDA memory allocator cannot always perfectly reuse the fragmented blocks from previous logits allocations, this leads to GPU memory fragmentation.

• The logits tensor frees its memory after each step.
• But the freed blocks are not guaranteed to be contiguous or align with the next allocation request.
• As a result, these fragmented regions cannot be fully reused by the next logits tensor, nor efficiently used by other tensors.

When max_split_size_mb is set large enough (≥ logits tensor size), the allocator can place each logits tensor in a single contiguous block. Then, the same memory region can be reused across steps, avoiding fragmentation and stabilizing memory usage.

Start by tracing the ray submit launch configuration and the torch_memory_saver setting for 80GB GPU runs. Compare the reported logits-size formula and Qwen3 example with the current allocation behavior; done means the launch configuration applies an appropriate PYTORCH_CUDA_ALLOC_CONF value and avoids the reported CUDA OOM or fragmentation case.