Do you have any examples of PPOs?

Start with train.py at train() and the MegatronTrainRayActor.train_actor path in slime/backends/megatron_utils/actor.py, especially the traceback locations around lines 359 and 430. Reproduce the supplied PPO launch command and trace when _actor_critic_groups should be available; done means the PPO example runs past actor training without this AttributeError and the example or required setup is documented.

I used the following .sh file to run PPO

CKPT_ARGS=(
   --hf-checkpoint GLM/GLM-Z1-9B-0414/
   --ref-load GLM/GLM-Z1-9B-0414_torch_dist/
   --load ckpt/GLM-Z1-9B-0414_slime/
   --save ckpt/GLM-Z1-9B-0414_slime/
   --save-interval 20
)

ROLLOUT_ARGS=(
   --prompt-data data/dapo-math-17k/dapo-math-17k.jsonl
   --input-key prompt
   --label-key label
   --apply-chat-template

   --rollout-shuffle

   --rm-type deepscaler

   --num-rollout 3000

   --rollout-batch-size 16 
   --n-samples-per-prompt 8

   --global-batch-size 128        
   --num-steps-per-rollout 1

   --rollout-max-response-len 8192
   --rollout-temperature 0.8

   --balance-data
)

PERF_ARGS=(
   --tensor-model-parallel-size 2
   --sequence-parallel
   --pipeline-model-parallel-size 1
   --context-parallel-size 1
   --expert-model-parallel-size 1
   --expert-tensor-parallel-size 1

   --recompute-granularity full
   --recompute-method uniform
   --recompute-num-layers 1

   # --micro-batch-size 1
   --use-dynamic-batch-size
   --max-tokens-per-gpu 4608
)

PPO_ARGS=(
   --advantage-estimator ppo
   --use-kl-loss
   --kl-loss-coef 0.00
   --kl-loss-type low_var_kl
   --entropy-coef 0.00
   --eps-clip 0.2
   --eps-clip-high 0.28
   --use-tis
)

OPTIMIZER_ARGS=(
   --optimizer adam
   --lr 1e-6
   --lr-decay-style constant
   --weight-decay 0.1
   --adam-beta1 0.9
   --adam-beta2 0.98
)

SGLANG_ARGS=(
   --rollout-num-gpus-per-engine 2
)

MISC_ARGS=(
   # default dropout in megatron is 0.1
   --attention-dropout 0.0
   --hidden-dropout 0.0
   # should be good for model performance
   --accumulate-allreduce-grads-in-fp32
   --attention-softmax-in-fp32
   # need to comment this when using model with MLA
   --attention-backend flash
)

# launch the master node of ray in container
export MASTER_ADDR=${MASTER_ADDR:-"127.0.0.1"}
ray start --head --node-ip-address ${MASTER_ADDR} --num-gpus 8 --disable-usage-stats --dashboard-host=0.0.0.0 --dashboard-port=8265

# Build the runtime environment JSON with proper variable substitution
MEGATRON_LM_PATH="slime_project/Megatron-LM"
RAY_PYTHONPATH="${MEGATRON_LM_PATH}:${PYTHONPATH:-}"
RUNTIME_ENV_JSON="{
  \"env_vars\": {
    \"PYTHONPATH\": \"${RAY_PYTHONPATH}\",
    \"CUDA_DEVICE_MAX_CONNECTIONS\": \"1\",
    \"NCCL_NVLS_ENABLE\": \"${HAS_NVLINK}\"
  }
}"

CRITIC_ARGS=(  
   --critic-num-nodes 1  
   --critic-num-gpus-per-node 2  
)

ray job submit --address="http://127.0.0.1:8265" \
   --runtime-env-json="${RUNTIME_ENV_JSON}" \
   -- python3 train.py \
   --actor-num-nodes 1 \
   --actor-num-gpus-per-node 4 \
   --rollout-num-gpus 2 \
   ${MODEL_ARGS[@]} \
   ${CKPT_ARGS[@]} \
   ${ROLLOUT_ARGS[@]} \
   ${OPTIMIZER_ARGS[@]} \
   ${PPO_ARGS[@]} \
   ${WANDB_ARGS[@]} \
   ${PERF_ARGS[@]} \
   ${EVAL_ARGS[@]} \
   ${SGLANG_ARGS[@]} \
   ${MISC_ARGS[@]} \
   ${CRITIC_ARGS[@]} \
   --partial-rollout

However, the following error occurred:

Traceback (most recent call last):
  File "/home/zzli/zxz/slime_project/slime/train.py", line 106, in <module>
    train(args)
  File "/home/zzli/zxz/slime_project/slime/train.py", line 76, in train
    ray.get(actor_model.async_train(rollout_id, rollout_data_ref))
  File "/root/miniconda3/envs/slime/lib/python3.12/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/slime/lib/python3.12/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/slime/lib/python3.12/site-packages/ray/_private/worker.py", line 2967, in get
    values, debugger_breakpoint = worker.get_objects(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/slime/lib/python3.12/site-packages/ray/_private/worker.py", line 1015, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(AttributeError): ray::MegatronTrainRayActor.train() (pid=1980964, ip=job-677c4fc6-379b-472c-abe9-c07b6611a286-master-0, actor_id=7164654a059b821e918d082902000000, repr=<slime.backends.megatron_utils.actor.MegatronTrainRayActor object at 0x7f3ad972ea50>)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zzli/zxz/slime_project/slime/slime/backends/megatron_utils/actor.py", line 359, in train
    return self.train_actor(rollout_id, rollout_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zzli/zxz/slime_project/slime/slime/backends/megatron_utils/actor.py", line 430, in train_actor
    self._actor_critic_groups,
    ^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'MegatronTrainRayActor' object has no attribute '_actor_critic_groups'

I noticed that the error occurred after rollout and actor model.