vLLM reasoning-parser path drops reasoning_content, so strip_thinking and logprob normalization cannot see the trace
- 主要語言
- Python
- 星號
- 70
- 分支
- 21
- 平均合併
- 2 天 11 小時
- 30 天內合併 PR
- 41
描述
Claude Code · https://claude.ai/code/session_01LMFjgUqHjQA3aCeB4QmEHZ — from gregorybchris's agent review of #361 (posted on #363), verified in code.
`vllm_server` reads `choice.message.content` only (`inference/providers/vllm_server.py` ~L935). With `--reasoning-parser` the trace arrives in `reasoning_content` and is discarded: no `` remains, `strip_thinking` no-ops, `original_text` is never recorded, and logprobs span the full generation while byte/char counts cover only the answer — the skew #361 fixed for the tag-in-text case.
Parity angle: oe-eval scores raw completions after `r1_style`, keeping the post-trace text byte-for-byte (see #363 for why leading whitespace matters to IFEval). A parser-produced `content` may already be trimmed, so parser-path runs can diverge from oe-eval even with `strip_thinking` on.
Suggested fix: keep `reasoning_content` — prepend it (wrapped in `…`) into the text before `strip_thinking`, or add a first-class reasoning field on `LMOutput` that the strip and the normalizers consult. Until then, run parity evals without a reasoning parser.
貢獻指南
這個儲存庫沒有索引到貢獻指南
研究方向
Start in inference/providers/vllm_server.py around line 935 and trace how reasoning-parser responses become LMOutput text. Inspect strip_thinking, original_text handling, logprob normalization, and the r1_style path, then compare their treatment of reasoning_content with the behavior described in #361 and #363. Done means parser-path outputs preserve the reasoning trace for stripping and keep text, byte/character, and logprob accounting aligned with oe-eval.
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- python
- 領域
- ai, backend
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 活躍
- 描述清晰度
- 基本清楚
- 新手友好度
- 68/100