[Question] Would an adaptive in-reward KL controller fit Slime's PPO/RLHF scope?

Your Question

Question

Would an opt-in adaptive in-reward KL controller for PPO-style reference-policy training fit Slime's current scope for general-purpose RL optimizations?

Proposal

The proposal is deliberately limited to the existing PPO/RLHF-style reward-side KL path:

observed reference KL -> update beta -> next-batch reward shaping
r'_t = task_reward_t - beta * KL_t

The controller would adapt the existing --kl-coef between batches toward a target KL, with bounded updates, a configurable horizon, checkpointed state, and metrics for observed KL, target KL, coefficient, and update status. The default behavior would remain fixed-coefficient and unchanged.

This would run alongside PPO clipping; it would not modify eps_clip or the PPO ratio objective.

Scope boundary

This is not a proposal to change the current GRPO reward semantics. I understand the maintainer direction in #399: GRPO should prefer kl_loss unless experiments show that reward-side KL is beneficial. The first implementation would target PPO-style training only, where reward-side KL is already part of the existing path.

Would a focused implementation with unit tests and a reproducible training benchmark be considered an in-scope general-purpose RL optimization? If so, I would be happy to prepare a small PR rather than a broad abstraction proposal.

What I've Tried

I checked the current Slime implementation and related discussions:

• #397 identified that GRPO's non-zero --kl-coef was effectively ignored.
• #399 proposed applying KL to GRPO rewards, but the maintainer discussion preferred kl_loss for GRPO unless supported by experiments.
• #1247 made a similar GRPO reward-side change and was closed without being merged.
• In the current PPO path, --kl-coef is applied during token-level reward construction before GAE.
• In verl's implementation, AdaptiveKLController updates the coefficient from the batch KL toward a target KL. The implementation cites Ziegler et al.'s RLHF work (arXiv:1909.08593).

I have intentionally not included GRPO, TTTD, OPD, standalone kl_loss_coef, or evaluation/rollback orchestration in this proposal. Those have different semantics and should be considered separately.

Environment (if relevant)

This is an algorithm/API-scope question rather than a runtime bug. No specific hardware or environment is required at this stage.

Additional Context

Related discussion: Human-Agent-Society/reef#466.

The goal is to clarify ownership and scope before writing code. Slime would own the training-side controller; any held-out evaluation or publish/pause/rollback policy would remain outside this proposed Slime change.

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and FAQ and my question is not answered there.
• I have searched for existing issues and my question has not been asked before.

Start by reviewing Slime's current PPO token-level reward construction and existing --kl-coef path described in the issue, then compare the requested behavior with verl's AdaptiveKLController. A completed implementation would need bounded cross-batch updates, checkpointed state, metrics, unit tests, and a reproducible PPO benchmark while leaving defaults and GRPO semantics unchanged.