google / google/adk-python

LLM-as-judge default config triggers google-genai's AFC warning in every eval run

Offen Anfängerfreundlich
#7,146 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
21.5k
Forks
4k
Ø Merge
1 T. 14 Std.
Gemergte PRs (30 T.)
37

Beschreibung

## Summary

Every rubric-based eval run logs this google-genai warning once per process:

```
WARNING google_genai.models: Direct use of automatic function calling (AFC) in AsyncModels.generate_content is not recommended. Instead, we recommend to use AFC in AsyncChat.send_message. Similarly, direct use of AFC in AsyncModels.generate_content_stream is not recommended. ...
```

The agent under evaluation does not cause it. It comes from the LLM-as-judge request.

## Where it comes from

`LlmAsJudge` builds the judge request with `config=self._judge_model_options.judge_model_config or genai_types.GenerateContentConfig()` (`src/google/adk/evaluation/llm_as_judge.py:224-225` on `main` at `ce53a36c0a`, and the same in 2.9.0 and 2.9.1).

That default config sets neither `tools` nor `automatic_function_calling`. google-genai's `AsyncModels.generate_content` therefore takes its AFC branch and logs the warning:
- `_extra_utils.should_disable_afc` returns `False` when `automatic_function_calling` is unset.
- With no tools there are no AFC-incompatible tool indexes, so the direct `_generate_content` path is skipped.

An agent's own model calls do not hit this. ADK sends tools as `function_declarations`, which google-genai marks AFC-incompatible, so those calls go straight to `_generate_content`.

## How it was observed

In an eval suite running `rubric_based_*_quality_v1` metrics with `gemini-3.5-flash` as judge (google-adk 2.9.0, google-genai as resolved by it):
- The warning appears once per pytest process.
- It always appears right after the agent's inference for the case ends and right before the rubric verdicts.
- It never appears during the agent's own model calls.
- The deployed agent's logs, same code without the eval harness, contain no occurrence over 7 days.

## Suggestion

The judge never needs automatic function calling. Its default config could disable it explicitly:

```python
config=self._judge_model_options.judge_model_config
or genai_types.GenerateContentConfig(
automatic_function_calling=genai_types.AutomaticFunctionCallingConfig(disable=True)
),
```

This does not change the request sent to the model (the field is client-side only) and removes a warning that points users at their own agent. Users who pass `judge_model_config` are unaffected.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start at src/google/adk/evaluation/llm_as_judge.py:224-225, where LlmAsJudge creates the default GenerateContentConfig for judge requests. Verify the default disables automatic function calling without changing user-supplied judge_model_config, then run the relevant evaluation checks and confirm the google-genai AFC warning no longer appears.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
testing-qa
Issue-Typ
Bug
Schwierigkeit
1/5
Geschätzter Aufwand
1-3 Stunden
Aktivitätsstatus
Aktiv
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
86/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.