allenai / allenai/reward-bench

The `Atla` inference branch calls an argument `vllm==0.13.0` does not accept

未关闭
#276 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
736
派生
99
PR 合并指标
30 天内没有已合并 PR

描述

Filed separately from #275, which fixes the context-length flag in the same two files. This
one is an issue rather than a pull request because the fix depends on how the Atla models are
meant to be tokenised, and that is a maintainer's call.

Environment: `05a9005efb607249822c193590c8ecab87c77052`, `vllm==0.13.0` as pinned in
`pyproject.toml:72`. Line numbers read with `git show 05a9005:`.

## The call

`scripts/run_generative_v2.py:601` and `scripts/run_generative.py:350`:

```python
if model_modifier == "Atla":
logger.info("Using Atla model for inference")
outputs = model.generate(prompt_token_ids=prompt_ids, sampling_params=sampling_params)
```

`LLM.generate` in the pinned version takes `(self, prompts, sampling_params, use_tqdm,
lora_request, priority)`. There is no `prompt_token_ids`:

```
$ python -c "import inspect, vllm; print(vllm.__version__, list(inspect.signature(vllm.LLM.generate).parameters))"
0.13.0 ['self', 'prompts', 'sampling_params', 'use_tqdm', 'lora_request', 'priority']
```

So any model routed through the branch raises `TypeError` at inference — after the weights
have loaded, not at argument parsing. Observed with `AtlaAI/Selene-1-Mini-Llama-3.1-8B`,
which is the model the branch exists for. The `prompt_ids` column is built for this path and
consumed nowhere else.

## Reproducing

```bash
git clone https://github.com/allenai/reward-bench.git && cd reward-bench
git checkout 05a9005efb607249822c193590c8ecab87c77052
pip install -e ".[vllm]" google-genai together # the last two are needed to import; see #274
python scripts/run_generative_v2.py --model AtlaAI/Selene-1-Mini-Llama-3.1-8B
```

## Why no PR

Passing token ids to a vLLM version that does not take them can be resolved more than one
way — detokenise and use the `prompts` path, or pass `TokensPrompt` objects — and which is
right depends on whether the branch exists to bypass the chat template or to control special
tokens. We dropped that judge from our own screen and reported the count rather than guess.

Happy to send a PR once the intended behaviour is settled.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。