aws-samples / aws-samples/sample-gen-ai-evaluations-workshop

Seven notebooks grade with the same model they evaluate; one says a stronger model is judging

Open
#90 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
70
Forks
27
Avg merge
8d 20h
Merged PRs (30d)
5

Description

Read at `4a72984`. Most of this repo does this well — the `AGENT_MODEL` / `JUDGE_MODEL` two-constant pattern is used in nine judge-bearing files, and `02-quality-metrics/03_Evaluating_your_Judge.ipynb` is a real validation step. This issue is about the seven files (of 37 that grade) that don't follow the repo's own pattern.

**1. The entry module aliases judge to subject, and one notebook says otherwise**

`Foundational Evaluations/model_config.py:25` — `JUDGE_MODEL_ID = DEFAULT_MODEL_ID`. The comment above it (L23–24) says *"Kept separate so you can use a stronger (or cheaper) judge without changing the model under evaluation"* — but as shipped they are the same literal. Downstream:

- `02-quality-metrics/utils.py:185` `call_judge_model(prompt, model_id=JUDGE_MODEL_ID)`; every caller omits the argument (L341), so sonnet-5 output is graded by sonnet-5. `01_LLM_as_Judge_analysis.ipynb` inherits this via `from utils import *`.
- `04-agentic-metrics/01-Agentic-Metrics.ipynb` cell 35: *"using a stronger model to evaluate responses from a weaker model"*; cell 36 `# Create an evaluator agent with a stronger model` → `model=JUDGE_MODEL_ID`. As shipped, that is the same model. The prose is false.

**2. One variable, both roles**

- `03-understanding-failures/01_Discovering_Failure_Patterns.ipynb` — one `bedrock_model` (cell 4) is the persona agent (59), the replay agent (61), and `judge_agent` (68).
- `Industry Specific Evaluations/Retail and CPG/retail_cpg_evaluations.ipynb` — one `MODEL_ID` behind one `invoke_model()`; cell 4 generates the product description, cell 8 `JUDGE_PROMPT` grades it for factual accuracy and hallucination, cell 13 `REASONING_JUDGE` again. The model checks its own output for hallucination.
- `Workload Specific Evaluations/Basic RAG/example-notebook/01 Basic RAG Evaluation.ipynb` cell 34 — one `HAIKU_ID` passed to both `RAGClient(...)` and `EvaluationClient(...)`.

**3. Inherited silently**

- `Framework Specific Evaluations/DSPy/01 DSPy Prompt Optimization.ipynb` — cell 5 sets one global LM; cell 41 `faithfulness_judge = dspy.Predict(FaithfulnessCheck)` with no `lm=`, so the judge inherits the subject's model and its score feeds the optimizer.

None of the seven says so in a comment or README.

**Suggested fix**

Give `JUDGE_MODEL_ID` a distinct default (or make the cell-35/36 wording conditional); add a judge constant to Retail/CPG, Basic RAG and 03-understanding-failures; pass `lm=` to the DSPy judge. Where sharing a model is intended, a one-line comment is enough — `03_Evaluating_your_Judge.ipynb` already makes the case for why it matters.

**How I found them**

By classifying each file by which model *generates* and which *grades*, not by variable name. A grep for `JUDGE_MODEL` matches exactly the files that do it right and misses every one above.

Contributor guide

Open the contributing guide

Research direction

Read commit 4a72984, then compare the seven named notebooks and model_config.py with 02-quality-metrics/03_Evaluating_your_Judge.ipynb and the nine files using separate model constants. Trace each subject and judge entry point, including utils.py:185 and the cited notebook cells. Done means each shared model is separated or explicitly documented as intentional, with the prose and evaluator wiring matching the actual roles.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
ai, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.