huggingface / huggingface/lighteval

[BUG] LLM judge silently scores API error strings: litellm backend sends max_tokens as a list, provider returns 400

Open Beginner friendly
#1,296 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug

When a judge is constructed with `max_tokens` set, the `litellm` backend sends `max_tokens` as a **one-element list** instead of an integer. Any spec-compliant OpenAI-compatible server rejects the request with `400 Bad Request`. `lighteval` then retries 3x, gives up, and returns the string **as the judge's response**. That string is passed straight into `process_judge_response()` and scored as if it were a real judgment. No crash, no warning, just corrupted judge scores.

**Root cause** — `src/lighteval/metrics/utils/llm_as_judge.py`, in `JudgeLM.__call_litellm` (line **341** on `main`, line **331** in the released 0.13.0):

```python
if max_new_tokens is not None:
kwargs["max_tokens"] = (max_new_tokens,) # trailing comma -> 1-tuple
```

Serialized to JSON, that becomes `"max_tokens": [64]`.

Note:
1. This only fires when `max_tokens` is explicitly set on the judge.
2. Only the `litellm` backend is affected.

## To Reproduce

Against any OpenAI-compatible server. I used vLLM:

```bash
vllm serve Qwen/Qwen3.6-27B-FP8 \
--api-key sk-fake
```

```python
import os

VLLM = "http://localhost:8000/v1"
os.environ["OPENAI_BASE_URL"] = VLLM
os.environ["OPENAI_API_KEY"] = "sk-fake"

import litellm
from lighteval.metrics.utils.llm_as_judge import JudgeLM

# print what lighteval hands to litellm
real_completion = litellm.completion
def spy(**kwargs):
print("max_tokens passed to litellm:", repr(kwargs["max_tokens"]))
return real_completion(**kwargs)
litellm.completion = spy

judge = JudgeLM(
model="openai/Qwen/Qwen3.6-27B-FP8",
templates=lambda question, answer, options=None, gold=None, **kw: [
{"role": "user", "content": question}
],
process_judge_response=lambda r: r,
judge_backend="litellm",
max_tokens=64,
backend_options={"caching": False},
)

_, _, responses = judge.evaluate_answer_batch(
questions=["Explain step by step why 2 + 2 = 4."],
answers=["4"], options=[None], golds=["4"],
)
print("judge response:", responses[0])

# control: identical call, max_tokens as the int it was meant to be
ok = real_completion(
model="openai/Qwen/Qwen3.6-27B-FP8",
messages=[{"role": "user", "content": "Explain step by step why 2 + 2 = 4."}],
max_tokens=64,
)
print("control ok:", ok.choices[0].message.content[:40])
```

**Output:**

```
max_tokens passed to litellm: (64,)
judge response: ERROR: Failed to get response from the API.
control ok: ...
```

**vLLM's access log** — same server, same model, same prompt; the only difference is the tuple:

```
POST /v1/chat/completions HTTP/1.1" 400 Bad Request <- judge, attempt 1
POST /v1/chat/completions HTTP/1.1" 400 Bad Request <- judge, attempt 2 (3x retry)
POST /v1/chat/completions HTTP/1.1" 400 Bad Request <- judge, attempt 3
POST /v1/chat/completions HTTP/1.1" 200 OK <- control (max_tokens=64 as int)
```

I also captured the raw request body by putting a recording server behind a real litellm proxy. Neither the litellm SDK nor the proxy validates or coerces the value and the malformed value goes to the provider

```json
{
"messages": [{"role": "user", "content": "..."}],
"model": "dummy-model",
"max_tokens": [512],
"n": 1
}
```

## Expected behavior

`max_tokens` should be sent as an integer, and the request should succeed:

```python
kwargs["max_tokens"] = max_new_tokens
```

With that one-line change, the same script returns `200 OK` from vLLM and the judge produces a real judgment instead of an error string.

## Version info

- OS: Linux
- lighteval: **0.13.0** (from PyPI) — also present on current `main` (`64f4f5ae`)
- Python 3.12, litellm 1.92.0
- Backend: `litellm`

Happy to open a PR with the fix.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/lighteval/metrics/utils/llm_as_judge.py at JudgeLM.__call_litellm, around line 341 on main, and inspect how max_new_tokens is placed in kwargs. Reproduce the litellm call against an OpenAI-compatible server or vLLM with max_tokens=64. Done means max_tokens is sent as an integer and the judge returns a real judgment instead of an API error string.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
1/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.