config option to disable monotonicity checks and on policy token id correction
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
**Is your feature request related to a problem? Please describe.**
Not all Nemo gym environments provide monotonic increasing trajectories (eg dropping reasoning in subsequent turns, multi-agent systems, etc). Currently, checks cause this to error. I would like a config parameter enforce_monotonicity to disable these checks. Along with this is replace_prefix_tokens, which could be disabled as a separate conflig flag, or bundled with this. I would like to disable this in the config as not all environments can propagate tokenids and logprobs for this (eg the current state of verifiers environments hub in nemo gym)
Monotonic check example: https://github.com/NVIDIA-NeMo/RL/blob/main/nemo_rl/models/generation/vllm/vllm_worker_async.py#L112
replace prefix tokens: https://github.com/NVIDIA-NeMo/RL/blob/main/nemo_rl/models/generation/vllm/vllm_worker_async.py#L40
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Assessment
This issue has not been assessed yet.