[Question] Multiple critic updates per actor update (SAO-style faster value update)?

Your Question

I'm trying to understand how the critic is trained relative to the actor in the PPO path, and I'm confused about why there doesn't seem to be a way to update the critic more times than the actor within one rollout. From what I can tell, train_critic and train_actor both call the same train(...) with the same num_microbatches / global_batch_sizes, so they run 1:1, and the only critic-related knob is --num-critic-only-steps, which is just a cross-rollout warmup.

I was reading the SAO paper (Single-Rollout Asynchronous Optimization, https://arxiv.org/abs/2607.07508, the method behind GLM-5.2's agentic RL) and noticed it uses a "faster value update" where the value network is updated K times per policy update (K > 1, paper uses K = 2) to reduce advantage-estimation variance when the critic lags behind the policy, with an ablation showing it helps. Since slime is the framework behind those runs, I was a bit surprised this isn't exposed in the open-source code.

So I'm wondering: is the current 1:1 behavior intentional, or would it make sense to add a config (something like --critic-update-steps / a critic-side K) to allow multiple critic updates per actor update? Raising this as a question first per CONTRIBUTING.md — thanks!

slime commit: 5b449de, PPO path (advantage_estimator=ppo, use_critic=True)

What I've Tried

I searched the codebase for any critic-specific update-count config (things like critic_iters, critic_epoch, critic_update, num_critic_steps) and couldn't find one — the only critic-related argument I found is --num-critic-only-steps, which only controls a cross-rollout warmup, not a per-rollout critic:actor update ratio.

I also traced the training path: both train_critic and train_actor in slime/backends/megatron_utils/actor.py call the same module-level train(...), and the schedule (num_microbatches / global_batch_sizes) comes from a single build_dp_schedule result shared by both, so as far as I can tell the critic and actor are structurally locked to the same number of updates per rollout.

Finally I read the SAO paper (arXiv:2607.07508) to confirm the "faster value update" (K=2) is an actual part of the recipe rather than a generic PPO trick, which is what prompted this question.

Environment (if relevant)

• slime version:
• Python version:
• PyTorch version:
• CUDA/ROCm version:
• GPU type and count:
• OS:

Additional Context

No response

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and FAQ and my question is not answered there.
• I have searched for existing issues and my question has not been asked before.

Start in slime/backends/megatron_utils/actor.py, comparing train_critic and train_actor and tracing the shared build_dp_schedule result. Read the handling of --num-critic-only-steps and the PPO path described in the issue. Done would require a maintainer decision on whether the 1:1 ratio is intentional or a scoped design for independent critic updates.