Qwen3-8B RL Training with Eagle3 Spec Decoding OOM error

Bug Description

When conducting RL training with Eagle3 speculative decoding on Qwen3-8B on 4*H200 (and even 8*H200), GPU OOM always occur after only 3~4 rollout steps.
According to the log, GPU memory usage keeps rising after each rollout step even though flush_cache and release_memory_occupation are called on the SGLang side. (An inspection of the full log indicates that, after each rollout step, the memory usage increases by about 15GB). It might be the cases that resume_memory_occupation allocates too much memory, or release_memory_occupation release too little, causing a memory leak across rollouts.
An excerpt of the OOM error is shown below:

(MegatronTrainRayActor pid=982439) [2025-12-03 06:17:51] timer.py:24 - Timer sleep start
(MegatronTrainRayActor pid=982439) [2025-12-03 06:17:52] memory_utils.py:41 - [Rank 0] Memory-Usage before offload model: {'gpu': '0', 'total_GB': 139.8, 'free_GB': 51.45, 'used_GB': 88.35, 'allocated_GB': 80.63, 'reserved_GB': 81.22}
(MegatronTrainRayActor pid=982439) [2025-12-03 06:17:54] memory_utils.py:41 - [Rank 0] Memory-Usage after offload model: {'gpu': '0', 'total_GB': 139.8, 'free_GB': 88.47, 'used_GB': 51.34, 'allocated_GB': 80.63, 'reserved_GB': 81.22}
(MegatronTrainRayActor pid=982439) [2025-12-03 06:17:54] timer.py:32 - Timer sleep end (elapsed: 2.4s)

(SGLangEngine pid=980268) [2025-12-03 06:17:55] INFO: 172.17.0.3:53854 - "POST /resume_memory_occupation HTTP/1.1" 200 OK
(SGLangEngine pid=980266) [2025-12-03 06:17:17] INFO: 172.17.0.3:40948 - "GET /health HTTP/1.1" 200 OK [repeated 3x across cluster]
(SGLangEngine pid=980269) [2025-12-03 06:17:55] Cache flushed successfully!
(SGLangEngine pid=980269) [2025-12-03 06:17:55] INFO: 172.17.0.3:48178 - "GET /flush_cache HTTP/1.1" 200 OK

(MegatronTrainRayActor pid=982439) [2025-12-03 06:17:55] timer.py:24 - Timer update_weights start
(MegatronTrainRayActor pid=982439) [2025-12-03 06:17:55] reloadable_process_group.py:145 - Reloading 7 process groups in pid 982439
(MegatronTrainRayActor pid=982439) [2025-12-03 06:17:55] memory_utils.py:41 - [Rank 0] Memory-Usage before update_weights: {'gpu': '0', 'total_GB': 139.8, 'free_GB': 70.92, 'used_GB': 68.88, 'allocated_GB': 80.63, 'reserved_GB': 81.22}

(MegatronTrainRayActor pid=982439) Update weights:   0%|          | 0/38 [00:00<?, ?it/s]
(SGLangEngine pid=980269) [2025-12-03 06:17:55] INFO: 172.17.0.3:48180 - "POST /update_weights_from_tensor HTTP/1.1" 200 OK
(SGLangEngine pid=980269) [2025-12-03 06:17:55] INFO: 172.17.0.3:48184 - "POST /update_weights_from_tensor HTTP/1.1" 200 OK

(MegatronTrainRayActor pid=982439) Update weights:   3%|▎         | 1/38 [00:00<00:10,  3.58it/s]
(MegatronTrainRayActor pid=982439) Update weights:  21%|██        | 8/38 [00:00<00:01, 25.95it/s]
(MegatronTrainRayActor pid=982439) Update weights:  39%|███▉      | 15/38 [00:00<00:00, 39.36it/s]
(MegatronTrainRayActor pid=982439) Update weights:  58%|█████▊    | 22/38 [00:00<00:00, 48.35it/s]
(MegatronTrainRayActor pid=982439) Update weights:  79%|███████▉  | 30/38 [00:00<00:00, 56.12it/s]
(MegatronTrainRayActor pid=982439) Update weights:  97%|█████████▋| 37/38 [00:00<00:00, 52.79it/s]
(MegatronTrainRayActor pid=982439) Update weights: 100%|██████████| 38/38 [00:00<00:00, 41.60it/s]

(MegatronTrainRayActor pid=982439) [2025-12-03 06:17:56] memory_utils.py:41 - [Rank 0] Memory-Usage after update_weights:
{'gpu': '0', 'total_GB': 139.8, 'free_GB': 49.36, 'used_GB': 90.44, 'allocated_GB': 95.89, 'reserved_GB': 101.91}

(SGLangEngine pid=980267) [torch_memory_saver.cpp] cuMemCreate CUDA_ERROR_OUT_OF_MEMORY (may not be an issue e.g. torch allocator will free cache and retry)
(SGLangEngine pid=980267) [torch_memory_saver.cpp] cudaError error: 2 (out of memory)  file=csrc/core.cpp func=resume line=182

(MegatronTrainRayActor pid=982439) [2025-12-03 06:17:56] timer.py:32 - Timer update_weights end (elapsed: 1.2s)

(SGLangEngine pid=980266) [2025-12-03 06:17:56] INFO: 172.17.0.3:54568 - "POST /resume_memory_occupation HTTP/1.1" 200 OK [repeated 10x across cluster]
(SGLangEngine pid=980269) [2025-12-03 06:18:17] INFO: 172.17.0.3:38206 - "GET /health HTTP/1.1" 200 OK
(SGLangEngine pid=980266) [2025-12-03 06:17:55] INFO: 172.17.0.3:54140 - "GET /flush_cache HTTP/1.1" 200 OK [repeated 3x across cluster]
(SGLangEngine pid=980266) [2025-12-03 06:17:56] INFO: 172.17.0.3:54546 - "POST /update_weights_from_tensor HTTP/1.1" 200 OK [repeated 150x across cluster]
(SGLangEngine pid=980267) [2025-12-03 06:18:17] INFO: 172.17.0.3:33086 - "GET /health HTTP/1.1" 200 OK
(SGLangEngine pid=980269) [2025-12-03 06:19:17] INFO: 172.17.0.3:41328 - "GET /health HTTP/1.1" 200 OK [repeated 3x across cluster]
(SGLangEngine pid=980269) [2025-12-03 06:20:17] INFO: 172.17.0.3:38886 - "GET /health HTTP/1.1" 200 OK [repeated 4x across cluster]

full error log:
run-qwen3-8b-rl-eagle3-OOM-issue.log

Environment

1. NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9, H200
2. image: slimerl/slime:latest

Steps for reproducing the bug

1. download models and dataset

hf download Qwen/Qwen3-8B --local-dir /root/Qwen3-8B
hf download Zjcxy-SmartAI/Eagle3-Qwen3-8B-zh --local-dir /root/Eagle3-Qwen3-8B-zh
hf download download --repo-type dataset zyzshishui0627/gsm8k --local-dir /root/gsm8k

2. convert model to torch_dist

cd /root/slime
source scripts/models/qwen3-8B.sh
PYTHONPATH=/root/Megatron-LM python tools/convert_hf_to_torch_dist.py \
    ${MODEL_ARGS[@]} \
    --hf-checkpoint /root/Qwen3-8B \
    --save /root/Qwen3-8B_torch_dist

3. run this script

cd /root/slime
bash ./scripts/run-qwen3-8b-rl-eagle3.sh

run-qwen3-8b-rl-eagle3.sh

Expected behavior

1. GPU memory usage should remain almost constant across rollout steps (memory usage before rollout starts and after rollout ends should be almost identical, rather than increasing).
2. RL training with Slime + SGLang + Eagle3 on Qwen3-8B should run without CUDA OOM on 4×H200 or 8×H200.

Start by reproducing with scripts/run-qwen3-8b-rl-eagle3.sh after following the model conversion steps in tools/convert_hf_to_torch_dist.py and scripts/models/qwen3-8B.sh. Compare the memory logs before and after each rollout, including the SGLang flush_memory and resume_memory_occupation calls shown in the report. Done means memory remains stable across rollouts and the Qwen3-8B Eagle3 run completes without CUDA OOM on the stated hardware.