allenai / allenai/reward-bench

rewardbench[vllm] cannot import run_generative_v2: google.genai and together are imported at module scope

Open
#274 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
736
Forks
99
PR merge metrics
No merged PRs in 30d

Description

`pyproject.toml` describes the `vllm` extra as "Local LLM inference via vLLM". Installing
exactly that and running the local script fails before any model loads.

### Reproduction

```
pip install "rewardbench[vllm] @ git+https://github.com/allenai/reward-bench.git@05a9005"
python -m scripts.run_generative_v2 --model Qwen/Qwen2.5-7B-Instruct --debug
```

```
File "rewardbench/generative_v2.py", line 27, in
from google import genai
ImportError: cannot import name 'genai' from 'google' (unknown location)
```

### Cause

`rewardbench/generative_v2.py` imports `google.genai` and `together` at module scope. The
`vllm` extra does not install either; `anthropic` and `openai` only import because vLLM
pulls them in transitively, so the failure surfaces on `google.genai` rather than earlier.

Installing `google-genai` and `together` fixes it, and neither is called on the local path.

### Possible direction

The repository already uses a guarded import for `fschat` in `scripts/run_v2.py`:

```python
try:
from fastchat.conversation import get_conv_template
except ImportError:
get_conv_template = None
```

The same pattern here, or moving the API client imports into the functions that use them,
would let the `vllm` extra stand on its own.

Happy to open a PR if that is useful.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.