GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK

Three examples pass one `MODEL_NAME` to both the agent and the AI.GENERATE judge; the SDK's judge defaults are silent

Open
#471 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
47
Forks
21
Avg merge
2d 13h
Merged PRs (30d)
33

Description

Read at `3222207`.

**What happens**

In three examples the variable that sets the agent's model is also the judge's model, so changing one moves the other:

- `examples/e2e_demo.py` — L74 `MODEL_NAME = os.environ.get("MODEL_NAME", "gemini-3-flash-preview")` → L322 `LlmAgent(model=MODEL_NAME)` → L439 `Client(endpoint=MODEL_NAME)` → L484 `LLMAsJudge.correctness()` → `client.evaluate`. The judge query is rendered with `endpoint=self.endpoint` (`src/bigquery_agent_analytics/client.py:2326-2332`), which is `MODEL_NAME`.
- `examples/context_graph_adcp_demo.ipynb` — same wiring: cell 6 `MODEL_NAME`, cell 10 `LlmAgent(model=MODEL_NAME)`, cell 15 `Client(endpoint=MODEL_NAME)`, cell 39 `client.evaluate(...)` (cells 0-indexed).
- `examples/nba_agent_trace_analysis_notebook.ipynb` — cell 4 `MODEL_NAME = "gemini-2.5-flash"` (no env override), cell 8 `LlmAgent(model=MODEL_NAME)`, cell 14 `LLMAsJudge.correctness(threshold=0.6, model=MODEL_NAME)`.

Nothing in those files or the README says the judge is the agent's model.

**Silent defaults**

With no endpoint or model passed, both grading paths use a hardcoded `"gemini-2.5-flash"`: `Client(endpoint=None)` → `DEFAULT_ENDPOINT` (`client.py:531`, `evaluators.py:52`) on the caller's BigQuery connection; `LLMAsJudge(model=None)` → `"gemini-2.5-flash"` (`evaluators.py:726`) via `genai.Client()` on ambient credentials. No warning on either path when that coincides with the agent's model.

**What this is not**

The repo also shows the right shape, which is why this is filed as three examples and not a design complaint: `examples/e2e_notebook_demo.ipynb` separates `AGENT_MODEL` from `SDK_ENDPOINT` (cells 5/8/12); `examples/skill_evolution_lab` has dedicated `AGENT_MODEL` / `ANALYST_MODEL` / `JUDGE_MODEL` with different defaults (`run_e2e_demo.sh`); `examples/agent_improvement_cycle` uses one `model_id` for both roles and `README.md:685` says so. No claim is made about score inflation — nothing was executed.

**Reproduce**

```
git checkout 3222207
grep -n 'MODEL_NAME' examples/e2e_demo.py # 74, 322, 439
grep -n 'DEFAULT_ENDPOINT\|or "gemini-2.5-flash"' src/bigquery_agent_analytics/evaluators.py src/bigquery_agent_analytics/client.py
```

**Suggested fix**

In the three examples, a second variable for the judge (`JUDGE_MODEL` / a separate `endpoint=`) defaulting to a different model, as `e2e_notebook_demo` already does. Optionally, one line in the `Client` and `LLMAsJudge` docstrings naming the default judge, and a note that `endpoint=MODEL_NAME` makes the agent grade itself.

Contributor guide

Open the contributing guide

Research direction

Compare the model separation in examples/e2e_notebook_demo.ipynb, then inspect the named model and client wiring in examples/e2e_demo.py, examples/context_graph_adcp_demo.ipynb, and examples/nba_agent_trace_analysis_notebook.ipynb. Check the defaults in src/bigquery_agent_analytics/client.py and evaluators.py and use the listed grep commands to verify the agent and judge can be configured independently.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.