Policy Collapse and Gibberish Rollouts in Search-R1 Early Training when TIS is Enabled (3B Model)

When reproducing Search-R1 using a 3B model on the Slime framework, I encountered a policy collapse during the early stages of training specifically when Token Importance Sampling (TIS) is enabled. I followed the tutorial settings in Enabling TIS (Trajectory Importance Sampling)

The model starts to generate chaotic, multilingual gibberish and completely ignores the system prompt and formatting constraints (e.g., and tags).

Comparison: TIS ON vs. OFF
TIS Enabled: The model collapses within the first few rollouts. Outputs become high-entropy "word salad" (as shown in the logs below).

TIS Disabled: Training is stable. The model follows the prompt instructions and converges normally.

In the early rollouts, the model fails to maintain the reasoning format and generate abnormal outputs:

Start with the Search-R1 tutorial's “Enabling TIS (Trajectory Importance Sampling)” settings and reproduce the 3B-model run in Slime with TIS enabled and disabled. Compare the early rollout logs and generated outputs, focusing on policy collapse, prompt adherence, and the / formatting. Done means the TIS-enabled run no longer produces gibberish or collapses during early rollouts.