NVIDIA-NeMo / NVIDIA-NeMo/RL

[Async PPO] cleanup tracking issue

Open
#3,695 1 comment 0 reactions 1 assignee Claimed by @terrykong View on GitHub
enhancement Feature
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

Tracking issue for loose ends in Async PPO, introduced in #3410.
Items will be checked off or split into separate issues as they land.

### Known missing items

- [ ] **The PPO branch of `AsyncTrajectoryCollector.__init__` hard-codes three settings that GRPO reads from config.** [`trajectory_collector.py:111-113`](https://github.com/NVIDIA-NeMo/RL/blob/bac4a8c46ee14d4c3dfb774be9577bacbb92ccf5/nemo_rl/algorithms/async_utils/trajectory_collector.py#L111-L113) sets `_deduplicate_multimodal_data = False`, `_debug_payload_metrics = False`, `_max_generation_failures = 0`, while the GRPO branch ten lines above reads all three from `GRPOConfig.deduplicate_multimodal_data`, `GRPOConfig.debug_payload_metrics`, and `AsyncGRPOConfig.max_generation_failures`. `PPOConfig` / `AsyncPPOConfig` carry no equivalent fields, so none of the three is reachable from a recipe. The `max_generation_failures = 0` one has runtime consequences: async PPO aborts on the first generation-worker exception with no way to opt into tolerating transient failures. Add the fields to `PPOConfig` / `AsyncPPOConfig` (mirroring the GRPO names and YAML docs) and read them here. (Flagged during #3410 review.)

- [ ] **Wire up (or keep fail-loud) the features `_validate_async_ppo_config` currently rejects.** [`run_ppo.py:48-88`](https://github.com/NVIDIA-NeMo/RL/blob/bac4a8c46ee14d4c3dfb774be9577bacbb92ccf5/examples/run_ppo.py#L48-L88) raises `NotImplementedError` for each of:
- `ppo.max_num_epochs != -1` — async PPO has no epoch limit; `max_num_steps` is the only length control. Same gap async GRPO has in #3175.
- NeMo-Gym rollout — the shared collector supports gym on the GRPO path, so this is a wiring gap rather than a missing capability.
- FP8 KV-scale synchronization — same gap tracked for the Single Controller in #2625; the driver-side compute + forward is what's missing.
- Dynamic sampling, reward scaling, reward shaping, multiple dataloaders.

Add anything else here as it comes up.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.