allenai / allenai/olmo-eval

vLLM reasoning-parser path drops reasoning_content, so strip_thinking and logprob normalization cannot see the trace

未關閉
#366 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
70
分支
21
平均合併
2 天 11 小時
30 天內合併 PR
41

描述

Claude Code · https://claude.ai/code/session_01LMFjgUqHjQA3aCeB4QmEHZ — from gregorybchris's agent review of #361 (posted on #363), verified in code.

`vllm_server` reads `choice.message.content` only (`inference/providers/vllm_server.py` ~L935). With `--reasoning-parser` the trace arrives in `reasoning_content` and is discarded: no `` remains, `strip_thinking` no-ops, `original_text` is never recorded, and logprobs span the full generation while byte/char counts cover only the answer — the skew #361 fixed for the tag-in-text case.

Parity angle: oe-eval scores raw completions after `r1_style`, keeping the post-trace text byte-for-byte (see #363 for why leading whitespace matters to IFEval). A parser-produced `content` may already be trimmed, so parser-path runs can diverge from oe-eval even with `strip_thinking` on.

Suggested fix: keep `reasoning_content` — prepend it (wrapped in `…`) into the text before `strip_thinking`, or add a first-class reasoning field on `LMOutput` that the strip and the normalizers consult. Until then, run parity evals without a reasoning parser.

貢獻指南

這個儲存庫沒有索引到貢獻指南

研究方向

Start in inference/providers/vllm_server.py around line 935 and trace how reasoning-parser responses become LMOutput text. Inspect strip_thinking, original_text handling, logprob normalization, and the r1_style path, then compare their treatment of reasoning_content with the behavior described in #361 and #363. Done means parser-path outputs preserve the reasoning trace for stripping and keep text, byte/character, and logprob accounting aligned with oe-eval.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
python
領域
ai, backend
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
活躍
描述清晰度
基本清楚
新手友好度
68/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。