[Question] OOM during long-sequence training？

Your Question

Training crash at step 75 with Qwen3-30B-A3B-Thinking-2507 (8×B200, 1600GB RAM) – suspected memory-related worker failure

error: The actor is dead because its worker process has died. Worker exit type: SYSTEM_ERROR
Worker exit detail: Worker exits unexpectedly. Worker exits with an exit code 1.
The worker may have exceeded K8s pod memory limits. The process receives a SIGTERM.

Core Parallelism

Tensor Parallelism (TP): 4

Pipeline Parallelism (PP): 1

Context Parallelism (CP): 1

Sequence Parallelism (SP): Enabled

MoE / Expert Parallelism

Expert Parallelism (EP): 4

The script is as follows:

#!/bin/bash


# for rerun the task
# pkill -9 sglang
# sleep 3
ray stop --force
pkill -9 ray
# pkill -9 python
# sleep 3
# pkill -9 ray
# pkill -9 python

set -ex

# will prevent ray from buffering stdout/stderr
export PYTHONUNBUFFERED=1

NVLINK_COUNT=$(nvidia-smi topo -m 2>/dev/null | grep -o 'NV[0-9][0-9]*' | wc -l)
if [ "$NVLINK_COUNT" -gt 0 ]; then
    HAS_NVLINK=1
else
    HAS_NVLINK=0
fi
echo "HAS_NVLINK: $HAS_NVLINK (detected $NVLINK_COUNT NVLink references)"

source "/scripts/models/qwen3-30B-A3B-thinking.sh"

# Generate timestamp suffix for save path


CKPT_ARGS=(
   --hf-checkpoint model/Qwen3-30B-A3B-Thinking-2507
   --ref-load model/Qwen3-30B-A3B-Thinking-2507_torch_dist
   --save Qwen3-30B_strands_${TIMESTAMP_SUFFIX}
   --save-interval 50
)

ROLLOUT_ARGS=(
   --prompt-data 
   --input-key prompt
   --label-key label
   --metadata-key metadata
   --rollout-shuffle
   # --reward-key score
   --num-rollout 300
   --rollout-batch-size 8
   --n-samples-per-prompt 8
   --rollout-max-response-len 32768
   --rollout-temperature 1

   --global-batch-size 64  
   --balance-data
)

EVAL_ARGS=(
   --eval-interval 5
   --eval-prompt-data 
   --n-samples-per-eval-prompt 1
   --eval-max-response-len 32768
   --eval-top-p 1
)

PERF_ARGS=(
   --tensor-model-parallel-size 4
   --sequence-parallel
   --pipeline-model-parallel-size 1
   --context-parallel-size 1
   --expert-model-parallel-size 4
   --expert-tensor-parallel-size 1

   --recompute-granularity full
   --recompute-method uniform
   --recompute-num-layers 1

   # --micro-batch-size 1
   --use-dynamic-batch-size
   --max-tokens-per-gpu 20480
)

GRPO_ARGS=(
   --advantage-estimator grpo
   --use-kl-loss
   --kl-loss-coef 0.00
   --kl-loss-type low_var_kl
   --entropy-coef 0.00
   --eps-clip 0.2
   --eps-clip-high 0.28
)

OPTIMIZER_ARGS=(
   --optimizer adam
   --lr 1e-6
   --lr-decay-style constant
   --weight-decay 0.1
   --adam-beta1 0.9
   --adam-beta2 0.98
)

WANDB_ARGS=(
   --use-wandb
   --wandb-project slime-dev
   --wandb-group qwen3-30B-A3B-thinking-2507-tool-nocolate-search
   --wandb-key ${WANDB_API_KEY}
)

SGLANG_ARGS=(
   --rollout-num-gpus-per-engine 4
   --sglang-mem-fraction-static 0.5
   #--sglang-log-level WARNING
   # Note: strands-sglang handles tool parsing internally (HermesToolCallParser)
   # No need for --sglang-tool-call-parser
)

MISC_ARGS=(
   # default dropout in megatron is 0.1
   --attention-dropout 0.0
   --hidden-dropout 0.0
   # should be good for model performance
   --accumulate-allreduce-grads-in-fp32
   --attention-softmax-in-fp32
   # need to comment this when using model with MLA
   --attention-backend flash
)

CUSTOM_ARGS=(
   --custom-generate-function-path examples.strands_sglang.generate_with_strands.generate
   --custom-rm-path examples.strands_sglang.reward.reward_func
)



# launch the master node of ray in container
export MASTER_ADDR=${MASTER_ADDR:-"127.0.0.1"}
ray start --head --node-ip-address ${MASTER_ADDR} --num-gpus 8 --disable-usage-stats --dashboard-host=0.0.0.0 --dashboard-port=8265

# Build the runtime environment JSON with proper variable substitution
RUNTIME_ENV_JSON="{
  \"env_vars\": {
    \"PYTHONPATH\": \"/root/Megatron-LM/:/root/slime\",
    \"CUDA_DEVICE_MAX_CONNECTIONS\": \"1\",
    \"NCCL_NVLS_ENABLE\": \"${HAS_NVLINK}\",
    \"TRAJECTORY_LOG_DIR\": \"${TRAJECTORY_LOG_DIR}\",
  }
}"

ray job submit --address="http://127.0.0.1:8265" \
   --runtime-env-json="${RUNTIME_ENV_JSON}" \
   -- python3 train_async.py \
   --actor-num-nodes 1 \
   --actor-num-gpus-per-node 4 \
   --rollout-num-gpus 4 \
   --distributed-timeout-minutes 30 \
   ${MODEL_ARGS[@]} \
   ${CKPT_ARGS[@]} \
   ${ROLLOUT_ARGS[@]} \
   ${OPTIMIZER_ARGS[@]} \
   ${GRPO_ARGS[@]} \
   ${WANDB_ARGS[@]} \
   ${PERF_ARGS[@]} \
   ${EVAL_ARGS[@]} \
   ${SGLANG_ARGS[@]} \
   ${MISC_ARGS[@]} \
   ${CUSTOM_ARGS[@]} 

Any suggestions will be greatly appreciated.

What I've Tried

I review all related issues and can not solve the problem.

Environment (if relevant)

• slime version:the latest
• Python version:3.12
• PyTorch version:12.9.1
• CUDA/ROCm version:12.8
• GPU type and count: 8 × NVIDIA B200
• OS:Linux
• Model: Qwen3-30b-a3b-thinking-2507

Additional Context

No response

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and FAQ and my question is not answered there.
• I have searched for existing issues and my question has not been asked before.

Start with train_async.py and the provided launch script, then inspect the Ray worker and Kubernetes memory details around the failure at step 75. Done means identifying the source of the worker termination and documenting a configuration or resource change that allows the long-sequence run to continue.