[Feature Request] Reward-hacking / training-instability monitoring transform

Motivation

Reward hacking / reward-model overoptimization is a common failure mode in RL post-training, and there's currently no standard TorchRL transform for monitoring it during a run.

Solution

An optional transform that tracks signals correlated with reward-hacking onset — KL-vs-reference acceleration, entropy-collapse trend, and advantage-distribution drift (Wasserstein vs. a rolling baseline) — and logs them alongside existing metrics. Related code: https://github.com/Aarav500/flight-recorder (Apache-2.0), which implements these extractors already (currently for GRPO/TRL-style training).

This comes from work on two reward-hacking-detection benchmarks (RHOB, Flight Recorder). One relevant finding: these signals reliably audit hacking after the fact but don't reliably give early warning at a usable false-positive rate (paper: "Audit or Early Warning? A Benchmark for Online Detectors of Reward Hacking") — worth knowing before assuming such a transform gives early warning rather than post-hoc detection.

Proposing before attempting a PR — happy to share methodology/code if there's interest.

Start by reviewing the proposed signals in this issue and the related Flight Recorder implementation, then compare them with the cited reward-hacking benchmark findings. Define the transform scope and metric behavior before implementation; done should mean an optional TorchRL transform that logs the agreed KL, entropy, and advantage-drift signals during training.