NVIDIA-NeMo / NVIDIA-NeMo/RL

Disallow truncate_history_thinking=true for NeMo-RL rollouts

Open
#3,931 0 comments 0 reactions 1 assignee Claimed by @terrykong View on GitHub
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

## Summary

`truncate_history_thinking=true` does not do what it is supposed to do in NeMo-RL rollouts, and it should not be supported as a rollout configuration.

We cannot simply truncate reasoning traces from previous turns in the current cumulative trajectory representation. Generation has seen those reasoning tokens, so training must see them too. Removing them only when constructing the training sequence would make the learner evaluate sampled tokens under a different causal context from the behavior policy.

Current NeMo-RL therefore preserves the reasoning:

- Native rollouts concatenate the exact generated token history;
- NeMo-Gym model servers restore the exact generated prefix after chat-template rendering.

This is necessary for correct on-policy training, but it means `truncate_history_thinking=true` is incompatible with the rollout contract.

The flag should not be repurposed as a context-compaction mechanism. Proper context compaction must be configured and represented explicitly, as in [PR #3910](https://github.com/NVIDIA-NeMo/RL/pull/3910), with exact per-call evidence and separate physical training traces.

## Why reasoning cannot simply be removed

Consider two policy calls:

```text
Call 1:
prompt = U1
response = R1 -> A1

Call 2 after removing reasoning history:
prompt = U1 -> A1 -> U2
response = R2 -> A2
```

Training `A1` must include `R1` in its causal context because `A1` was generated after and conditioned on `R1`.

If the second call genuinely uses the compacted prompt, `R2 -> A2` must instead be trained without `R1` in its causal context. These calls cannot be represented correctly as the single flat sequence:

```text
U1 -> R1 -> A1 -> U2 -> R2 -> A2
```

Changing context between calls requires separate physical traces containing the exact prompt tokens, sampled response tokens, and behavior-policy logprobs for each call. A model-specific Jinja chat-template option is not sufficient to declare or transport that training contract.

## Current behavior

### Native NeMo-RL rollout

Native multi-turn rollout does not reapply the chat template between policy calls. It stores the exact generated token IDs, appends environment tokens, and concatenates the complete token history for the next generation. Reasoning generated during an earlier call therefore remains in subsequent model inputs regardless of `truncate_history_thinking=true`.

Relevant code:

- `nemo_rl/experience/rollouts.py:542`
- `nemo_rl/experience/rollouts.py:931`
- `nemo_rl/experience/rollout_manager.py:587`

### NeMo-Gym rollout

NeMo-Gym may logically rerender the conversation without historical reasoning. NeMo-RL's model-serving layer then restores the exact previous prompt and generated-token prefix before the next generation. This prefix restoration preserves the on-policy cumulative trajectory, but also means reasoning removed by the template is added back to the actual model input.

Relevant code:

- `nemo_rl/models/generation/vllm/vllm_worker_async.py:411`
- `nemo_rl/models/generation/vllm/vllm_worker_async.py:508`
- `nemo_rl/models/generation/vllm/vllm_worker_async.py:818`
- `nemo_rl/models/generation/openai_server_utils.py:69`
- `nemo_rl/models/generation/openai_server_utils.py:104`

Equivalent prefix preservation exists in the TRT-LLM and Dynamo serving paths.

## Why this is a bug

NeMo-RL accepts `truncate_history_thinking=true` without explaining that the requested behavior is incompatible with cumulative RL rollouts.

A user can reasonably expect previous-turn reasoning to be absent from later model calls. In practice, native rollout retains it and NeMo-Gym serving restores it. The requested behavior is therefore silently unavailable.

This is not currently a training-correctness failure: NeMo-RL preserves the generated prefix specifically to maintain correct on-policy training. It is a configuration-semantics and observability bug.

## Expected behavior

For NeMo-RL rollout and training paths:

1. Reject an explicit `truncate_history_thinking=true` during configuration or startup.
2. Ensure cumulative rollouts preserve reasoning history, including when a model template would otherwise default to truncation.
3. Remove `truncate_history_thinking` as a user-selectable rollout option from NeMo-RL recipes once preservation is enforced internally.
4. Do not interpret this chat-template flag as a request for context compaction.
5. Direct users who need context removal to the explicit context-compaction rollout path and trace contract.
6. Add tests confirming that rollout configurations containing `true` fail with a clear error.

A possible error is:

```text
truncate_history_thinking=true is not supported for NeMo-RL rollouts.

NeMo-RL must preserve the exact causal context used to generate trainable
tokens. Use the explicit context-compaction rollout path when context must
change between model calls.
```

## Scope

This does not necessarily require deleting generic chat-template keyword support from tokenizers or offline independently rendered training pipelines.

The restriction is that `truncate_history_thinking=true` must not be accepted as NeMo-RL rollout behavior. Context compaction should retain its own explicit configuration, transport contract, validation, and physical-trace training representation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.