huggingface / huggingface/lighteval
[BUG] LLM judge silently scores API error strings: litellm backend sends max_tokens as a list, provider returns 400
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
When a judge is constructed with `max_tokens` set, the `litellm` backend sends `max_tokens` as a **one-element list** instead of an integer. Any spec-compliant OpenAI-compatible server rejects the request with `400 Bad Request`. `lighteval` then retries 3x, gives up, and returns the string **as the judge's response**. That string is passed straight into `process_judge_response()` and scored as if it were a real judgment. No crash, no warning, just corrupted judge scores.
**Root cause** — `src/lighteval/metrics/utils/llm_as_judge.py`, in `JudgeLM.__call_litellm` (line **341** on `main`, line **331** in the released 0.13.0):
```python
if max_new_tokens is not None:
kwargs["max_tokens"] = (max_new_tokens,) # trailing comma -> 1-tuple
```
Serialized to JSON, that becomes `"max_tokens": [64]`.
Note:
1. This only fires when `max_tokens` is explicitly set on the judge.
2. Only the `litellm` backend is affected.
## To Reproduce
Against any OpenAI-compatible server. I used vLLM:
```bash
vllm serve Qwen/Qwen3.6-27B-FP8 \
--api-key sk-fake
```
```python
import os
VLLM = "http://localhost:8000/v1"
os.environ["OPENAI_BASE_URL"] = VLLM
os.environ["OPENAI_API_KEY"] = "sk-fake"
import litellm
from lighteval.metrics.utils.llm_as_judge import JudgeLM
# print what lighteval hands to litellm
real_completion = litellm.completion
def spy(**kwargs):
print("max_tokens passed to litellm:", repr(kwargs["max_tokens"]))
return real_completion(**kwargs)
litellm.completion = spy
judge = JudgeLM(
model="openai/Qwen/Qwen3.6-27B-FP8",
templates=lambda question, answer, options=None, gold=None, **kw: [
{"role": "user", "content": question}
],
process_judge_response=lambda r: r,
judge_backend="litellm",
max_tokens=64,
backend_options={"caching": False},
)
_, _, responses = judge.evaluate_answer_batch(
questions=["Explain step by step why 2 + 2 = 4."],
answers=["4"], options=[None], golds=["4"],
)
print("judge response:", responses[0])
# control: identical call, max_tokens as the int it was meant to be
ok = real_completion(
model="openai/Qwen/Qwen3.6-27B-FP8",
messages=[{"role": "user", "content": "Explain step by step why 2 + 2 = 4."}],
max_tokens=64,
)
print("control ok:", ok.choices[0].message.content[:40])
```
**Output:**
```
max_tokens passed to litellm: (64,)
judge response: ERROR: Failed to get response from the API.
control ok: ...
```
**vLLM's access log** — same server, same model, same prompt; the only difference is the tuple:
```
POST /v1/chat/completions HTTP/1.1" 400 Bad Request <- judge, attempt 1
POST /v1/chat/completions HTTP/1.1" 400 Bad Request <- judge, attempt 2 (3x retry)
POST /v1/chat/completions HTTP/1.1" 400 Bad Request <- judge, attempt 3
POST /v1/chat/completions HTTP/1.1" 200 OK <- control (max_tokens=64 as int)
```
I also captured the raw request body by putting a recording server behind a real litellm proxy. Neither the litellm SDK nor the proxy validates or coerces the value and the malformed value goes to the provider
```json
{
"messages": [{"role": "user", "content": "..."}],
"model": "dummy-model",
"max_tokens": [512],
"n": 1
}
```
## Expected behavior
`max_tokens` should be sent as an integer, and the request should succeed:
```python
kwargs["max_tokens"] = max_new_tokens
```
With that one-line change, the same script returns `200 OK` from vLLM and the judge produces a real judgment instead of an error string.
## Version info
- OS: Linux
- lighteval: **0.13.0** (from PyPI) — also present on current `main` (`64f4f5ae`)
- Python 3.12, litellm 1.92.0
- Backend: `litellm`
Happy to open a PR with the fix.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/lighteval/metrics/utils/llm_as_judge.py at JudgeLM.__call_litellm, around line 341 on main, and inspect how max_new_tokens is placed in kwargs. Reproduce the litellm call against an OpenAI-compatible server or vLLM with max_tokens=64. Done means max_tokens is sent as an integer and the judge returns a real judgment instead of an API error string.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100