allenai / allenai/olmes

Unexpected keyword argument 'prompt_token_ids' for Gemma 3 Eval

未關閉
#32 5 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Python
星號
395
分支
105
PR 合併指標
30 天內沒有已合併 PR

描述

I'm running the "Olmes: Standard 10 MCQA" tasks using a VLLM backend and `gemma-3-1b-it`:

```bash
olmes \
--model google/gemma-3-1b-it \
--model-type vllm \
--model-args "trust_remote_code=True" \
--task \
core_9mcqa::olmes \
mmlu::olmes \
--batch-size auto \
--output-dir ./eval_results
```

However, I'm running into an error:

```
TypeError: LLM.generate() got an unexpected keyword argument 'prompt_token_ids'
[rank0]:[W1211 03:38:51.066672473 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
ERROR 12-11 03:38:51 [core_client.py:564] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.
Running loglikelihood requests: 0% 0/4000 [00:01

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。