NVIDIA-NeMo / NVIDIA-NeMo/RL

config option to disable monotonicity checks and on policy token id correction

Open
#1,812 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Is your feature request related to a problem? Please describe.**
Not all Nemo gym environments provide monotonic increasing trajectories (eg dropping reasoning in subsequent turns, multi-agent systems, etc). Currently, checks cause this to error. I would like a config parameter enforce_monotonicity to disable these checks. Along with this is replace_prefix_tokens, which could be disabled as a separate conflig flag, or bundled with this. I would like to disable this in the config as not all environments can propagate tokenids and logprobs for this (eg the current state of verifiers environments hub in nemo gym)

Monotonic check example: https://github.com/NVIDIA-NeMo/RL/blob/main/nemo_rl/models/generation/vllm/vllm_worker_async.py#L112

replace prefix tokens: https://github.com/NVIDIA-NeMo/RL/blob/main/nemo_rl/models/generation/vllm/vllm_worker_async.py#L40

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.