allenai / allenai/reward-bench
rewardbench[vllm] cannot import run_generative_v2: google.genai and together are imported at module scope
- Dominant language
- Python
- Stars
- 736
- Forks
- 99
- PR merge metrics
- No merged PRs in 30d
Description
`pyproject.toml` describes the `vllm` extra as "Local LLM inference via vLLM". Installing
exactly that and running the local script fails before any model loads.
### Reproduction
```
pip install "rewardbench[vllm] @ git+https://github.com/allenai/reward-bench.git@05a9005"
python -m scripts.run_generative_v2 --model Qwen/Qwen2.5-7B-Instruct --debug
```
```
File "rewardbench/generative_v2.py", line 27, in
from google import genai
ImportError: cannot import name 'genai' from 'google' (unknown location)
```
### Cause
`rewardbench/generative_v2.py` imports `google.genai` and `together` at module scope. The
`vllm` extra does not install either; `anthropic` and `openai` only import because vLLM
pulls them in transitively, so the failure surfaces on `google.genai` rather than earlier.
Installing `google-genai` and `together` fixes it, and neither is called on the local path.
### Possible direction
The repository already uses a guarded import for `fschat` in `scripts/run_v2.py`:
```python
try:
from fastchat.conversation import get_conv_template
except ImportError:
get_conv_template = None
```
The same pattern here, or moving the API client imports into the functions that use them,
would let the `vllm` extra stand on its own.
Happy to open a PR if that is useful.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.