[TRAIN-9] Thinking models (Qwen3/Qwen3.5) silently produce 0% accuracy due to low max_new_tokens
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
## Problem
Default `max_new_tokens` values in Qwen2.5 and Llama-3 GRPO recipes (3K-4K tokens) are too low for thinking models like Qwen3 and Qwen3.5. Reasoning traces exceed the limit and get truncated before the final `\boxed{N}` answer. Evaluation accuracy appears near 0% even though training loss and reward metrics look normal. There is no documentation warning about this when switching to thinking models.
## Steps to reproduce
1. Switch a GRPO config from a non-thinking baseline to Qwen3 or Qwen3.5.
2. Run training and observe evaluation accuracy near 0%.
3. Increase `max_new_tokens` in the vLLM config to >= 8K and confirm accuracy recovers.
## Suggested fix
- Add a documentation note in the Qwen recipe guides to set `max_new_tokens` to at least 8K for thinking models.
- Consider adding a config validation warning when a thinking model is detected but `max_new_tokens` is below a minimum threshold.
Contributor guide
Assessment
This issue has not been assessed yet.