vLLM reasoning-parser path drops reasoning_content, so strip_thinking and logprob normalization cannot see the trace
- Dominant language
- Python
- Stars
- 70
- Forks
- 21
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 41
Description
Claude Code · https://claude.ai/code/session_01LMFjgUqHjQA3aCeB4QmEHZ — from gregorybchris's agent review of #361 (posted on #363), verified in code.
`vllm_server` reads `choice.message.content` only (`inference/providers/vllm_server.py` ~L935). With `--reasoning-parser` the trace arrives in `reasoning_content` and is discarded: no `` remains, `strip_thinking` no-ops, `original_text` is never recorded, and logprobs span the full generation while byte/char counts cover only the answer — the skew #361 fixed for the tag-in-text case.
Parity angle: oe-eval scores raw completions after `r1_style`, keeping the post-trace text byte-for-byte (see #363 for why leading whitespace matters to IFEval). A parser-produced `content` may already be trimmed, so parser-path runs can diverge from oe-eval even with `strip_thinking` on.
Suggested fix: keep `reasoning_content` — prepend it (wrapped in `…`) into the text before `strip_thinking`, or add a first-class reasoning field on `LMOutput` that the strip and the normalizers consult. Until then, run parity evals without a reasoning parser.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.