[Async PPO] cleanup tracking issue
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
Tracking issue for loose ends in Async PPO, introduced in #3410.
Items will be checked off or split into separate issues as they land.
### Known missing items
- [ ] **The PPO branch of `AsyncTrajectoryCollector.__init__` hard-codes three settings that GRPO reads from config.** [`trajectory_collector.py:111-113`](https://github.com/NVIDIA-NeMo/RL/blob/bac4a8c46ee14d4c3dfb774be9577bacbb92ccf5/nemo_rl/algorithms/async_utils/trajectory_collector.py#L111-L113) sets `_deduplicate_multimodal_data = False`, `_debug_payload_metrics = False`, `_max_generation_failures = 0`, while the GRPO branch ten lines above reads all three from `GRPOConfig.deduplicate_multimodal_data`, `GRPOConfig.debug_payload_metrics`, and `AsyncGRPOConfig.max_generation_failures`. `PPOConfig` / `AsyncPPOConfig` carry no equivalent fields, so none of the three is reachable from a recipe. The `max_generation_failures = 0` one has runtime consequences: async PPO aborts on the first generation-worker exception with no way to opt into tolerating transient failures. Add the fields to `PPOConfig` / `AsyncPPOConfig` (mirroring the GRPO names and YAML docs) and read them here. (Flagged during #3410 review.)
- [ ] **Wire up (or keep fail-loud) the features `_validate_async_ppo_config` currently rejects.** [`run_ppo.py:48-88`](https://github.com/NVIDIA-NeMo/RL/blob/bac4a8c46ee14d4c3dfb774be9577bacbb92ccf5/examples/run_ppo.py#L48-L88) raises `NotImplementedError` for each of:
- `ppo.max_num_epochs != -1` — async PPO has no epoch limit; `max_num_steps` is the only length control. Same gap async GRPO has in #3175.
- NeMo-Gym rollout — the shared collector supports gym on the GRPO path, so this is a wiring gap rather than a missing capability.
- FP8 KV-scale synchronization — same gap tracked for the Single Controller in #2625; the driver-side compute + forward is what's missing.
- Dynamic sampling, reward scaling, reward shaping, multiple dataloaders.
Add anything else here as it comes up.
Contributor guide
Assessment
This issue has not been assessed yet.