[RFC] Support M2PO for stale-rollout GRPO

Checklist

• I have searched existing issues and pull requests, and this is a new feature request.

Feature Request Description

I would like to contribute support for M2PO (Second-Moment Trust Policy Optimization) to ms-swift's GRPO training stack.

M2PO was published at ICLR 2026. It targets off-policy RL with stale rollout data, which is directly relevant to the fully async rollout item in the current ms-swift roadmap. Instead of PPO's fixed token-wise clipping interval, the final paper's Algorithm 1 constrains the batch-level second moment of the policy/behavior log-ratio and masks only the largest outliers in the active trust-region quadrants.

Paper and reference implementation:

• Paper: https://arxiv.org/abs/2510.01161
• Reference repository: https://github.com/Infini-AI-Lab/M2PO

Proposed scope

1. Add m2po as a GRPO loss_type, with the paper default m2_threshold=0.04.
2. Compute log(pi_current / pi_behavior) from rollout_per_token_logps when rollout log-probabilities are available.
3. Apply M2 masking only to tokens in the two PPO clipping quadrants: (A > 0, ratio > 1) and (A < 0, ratio < 1).
4. Remove the largest squared log-ratio outliers until the second moment of the remaining trust-region tokens is at most the configured threshold.
5. Preserve the paper's denominator: average the masked objective over all valid completion tokens, not only the unmasked tokens.
6. Keep Transformers/HF and Megatron-SWIFT behavior aligned.
7. Add diagnostics such as m2_before, m2_after, masked_fraction, and trust_region_fraction.
8. Add deterministic unit tests for masking quadrants, threshold boundaries, padding/empty masks, numerical stability, and loss gradients, plus a small stale-rollout comparison against GRPO.

I already have an independent implementation of the final Algorithm 1 under review in AgentScope/Trinity-RFT#616. I would adapt the implementation and its edge-case tests to ms-swift's current GRPOBatch and distributed training abstractions rather than copying framework-specific code.

Before implementation, I would appreciate maintainer guidance on three details:

1. Should the initial PR include both the Transformers/HF and Megatron-SWIFT trainers, or land one backend first?
2. Should loss_type=m2po require rollout_per_token_logps, or fall back to old_per_token_logps when rollout log-probabilities are unavailable?
3. Should the M2 threshold be computed globally across data-parallel ranks, or per local training batch/micro-batch?

Pull Request

Yes. If this proposal fits the roadmap, I plan to submit the implementation, CPU unit tests, configuration/docs, and a reproducible small-model stale-rollout validation. PRs will be split if maintainers prefer separate backend or validation changes.

Start by tracing the GRPO loss_type path around GRPOBatch and rollout_per_token_logps in both the Transformers/HF and Megatron-SWIFT trainers. Run or add the proposed deterministic CPU unit tests for masking, boundaries, padding, numerical stability, gradients, and stale-rollout comparison. Done means m2po is configured, both backends align, diagnostics are exposed, and the tests and validation pass.