allenai / allenai/olmo-eval

vLLM reasoning-parser path drops reasoning_content, so strip_thinking and logprob normalization cannot see the trace

Open
#366 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
70
Forks
21
Avg merge
2d 11h
Merged PRs (30d)
41

Description

Claude Code · https://claude.ai/code/session_01LMFjgUqHjQA3aCeB4QmEHZ — from gregorybchris's agent review of #361 (posted on #363), verified in code.

`vllm_server` reads `choice.message.content` only (`inference/providers/vllm_server.py` ~L935). With `--reasoning-parser` the trace arrives in `reasoning_content` and is discarded: no `` remains, `strip_thinking` no-ops, `original_text` is never recorded, and logprobs span the full generation while byte/char counts cover only the answer — the skew #361 fixed for the tag-in-text case.

Parity angle: oe-eval scores raw completions after `r1_style`, keeping the post-trace text byte-for-byte (see #363 for why leading whitespace matters to IFEval). A parser-produced `content` may already be trimmed, so parser-path runs can diverge from oe-eval even with `strip_thinking` on.

Suggested fix: keep `reasoning_content` — prepend it (wrapped in `…`) into the text before `strip_thinking`, or add a first-class reasoning field on `LMOutput` that the strip and the normalizers consult. Until then, run parity evals without a reasoning parser.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.