GoogleCloudPlatform / GoogleCloudPlatform/BigQuery-Agent-Analytics-SDK
Three examples pass one `MODEL_NAME` to both the agent and the AI.GENERATE judge; the SDK's judge defaults are silent
- Dominant language
- Python
- Stars
- 47
- Forks
- 21
- Avg merge
- 2d 13h
- Merged PRs (30d)
- 33
Description
Read at `3222207`.
**What happens**
In three examples the variable that sets the agent's model is also the judge's model, so changing one moves the other:
- `examples/e2e_demo.py` — L74 `MODEL_NAME = os.environ.get("MODEL_NAME", "gemini-3-flash-preview")` → L322 `LlmAgent(model=MODEL_NAME)` → L439 `Client(endpoint=MODEL_NAME)` → L484 `LLMAsJudge.correctness()` → `client.evaluate`. The judge query is rendered with `endpoint=self.endpoint` (`src/bigquery_agent_analytics/client.py:2326-2332`), which is `MODEL_NAME`.
- `examples/context_graph_adcp_demo.ipynb` — same wiring: cell 6 `MODEL_NAME`, cell 10 `LlmAgent(model=MODEL_NAME)`, cell 15 `Client(endpoint=MODEL_NAME)`, cell 39 `client.evaluate(...)` (cells 0-indexed).
- `examples/nba_agent_trace_analysis_notebook.ipynb` — cell 4 `MODEL_NAME = "gemini-2.5-flash"` (no env override), cell 8 `LlmAgent(model=MODEL_NAME)`, cell 14 `LLMAsJudge.correctness(threshold=0.6, model=MODEL_NAME)`.
Nothing in those files or the README says the judge is the agent's model.
**Silent defaults**
With no endpoint or model passed, both grading paths use a hardcoded `"gemini-2.5-flash"`: `Client(endpoint=None)` → `DEFAULT_ENDPOINT` (`client.py:531`, `evaluators.py:52`) on the caller's BigQuery connection; `LLMAsJudge(model=None)` → `"gemini-2.5-flash"` (`evaluators.py:726`) via `genai.Client()` on ambient credentials. No warning on either path when that coincides with the agent's model.
**What this is not**
The repo also shows the right shape, which is why this is filed as three examples and not a design complaint: `examples/e2e_notebook_demo.ipynb` separates `AGENT_MODEL` from `SDK_ENDPOINT` (cells 5/8/12); `examples/skill_evolution_lab` has dedicated `AGENT_MODEL` / `ANALYST_MODEL` / `JUDGE_MODEL` with different defaults (`run_e2e_demo.sh`); `examples/agent_improvement_cycle` uses one `model_id` for both roles and `README.md:685` says so. No claim is made about score inflation — nothing was executed.
**Reproduce**
```
git checkout 3222207
grep -n 'MODEL_NAME' examples/e2e_demo.py # 74, 322, 439
grep -n 'DEFAULT_ENDPOINT\|or "gemini-2.5-flash"' src/bigquery_agent_analytics/evaluators.py src/bigquery_agent_analytics/client.py
```
**Suggested fix**
In the three examples, a second variable for the judge (`JUDGE_MODEL` / a separate `endpoint=`) defaulting to a different model, as `e2e_notebook_demo` already does. Optionally, one line in the `Client` and `LLMAsJudge` docstrings naming the default judge, and a note that `endpoint=MODEL_NAME` makes the agent grade itself.
Contributor guide
Research direction
Compare the model separation in examples/e2e_notebook_demo.ipynb, then inspect the named model and client wiring in examples/e2e_demo.py, examples/context_graph_adcp_demo.ipynb, and examples/nba_agent_trace_analysis_notebook.ipynb. Check the defaults in src/bigquery_agent_analytics/client.py and evaluators.py and use the listed grep commands to verify the agent and judge can be configured independently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100