aws-samples / aws-samples/sample-gen-ai-evaluations-workshop
Seven notebooks grade with the same model they evaluate; one says a stronger model is judging
- Dominant language
- Jupyter Notebook
- Stars
- 70
- Forks
- 27
- Avg merge
- 8d 20h
- Merged PRs (30d)
- 5
Description
Read at `4a72984`. Most of this repo does this well — the `AGENT_MODEL` / `JUDGE_MODEL` two-constant pattern is used in nine judge-bearing files, and `02-quality-metrics/03_Evaluating_your_Judge.ipynb` is a real validation step. This issue is about the seven files (of 37 that grade) that don't follow the repo's own pattern.
**1. The entry module aliases judge to subject, and one notebook says otherwise**
`Foundational Evaluations/model_config.py:25` — `JUDGE_MODEL_ID = DEFAULT_MODEL_ID`. The comment above it (L23–24) says *"Kept separate so you can use a stronger (or cheaper) judge without changing the model under evaluation"* — but as shipped they are the same literal. Downstream:
- `02-quality-metrics/utils.py:185` `call_judge_model(prompt, model_id=JUDGE_MODEL_ID)`; every caller omits the argument (L341), so sonnet-5 output is graded by sonnet-5. `01_LLM_as_Judge_analysis.ipynb` inherits this via `from utils import *`.
- `04-agentic-metrics/01-Agentic-Metrics.ipynb` cell 35: *"using a stronger model to evaluate responses from a weaker model"*; cell 36 `# Create an evaluator agent with a stronger model` → `model=JUDGE_MODEL_ID`. As shipped, that is the same model. The prose is false.
**2. One variable, both roles**
- `03-understanding-failures/01_Discovering_Failure_Patterns.ipynb` — one `bedrock_model` (cell 4) is the persona agent (59), the replay agent (61), and `judge_agent` (68).
- `Industry Specific Evaluations/Retail and CPG/retail_cpg_evaluations.ipynb` — one `MODEL_ID` behind one `invoke_model()`; cell 4 generates the product description, cell 8 `JUDGE_PROMPT` grades it for factual accuracy and hallucination, cell 13 `REASONING_JUDGE` again. The model checks its own output for hallucination.
- `Workload Specific Evaluations/Basic RAG/example-notebook/01 Basic RAG Evaluation.ipynb` cell 34 — one `HAIKU_ID` passed to both `RAGClient(...)` and `EvaluationClient(...)`.
**3. Inherited silently**
- `Framework Specific Evaluations/DSPy/01 DSPy Prompt Optimization.ipynb` — cell 5 sets one global LM; cell 41 `faithfulness_judge = dspy.Predict(FaithfulnessCheck)` with no `lm=`, so the judge inherits the subject's model and its score feeds the optimizer.
None of the seven says so in a comment or README.
**Suggested fix**
Give `JUDGE_MODEL_ID` a distinct default (or make the cell-35/36 wording conditional); add a judge constant to Retail/CPG, Basic RAG and 03-understanding-failures; pass `lm=` to the DSPy judge. Where sharing a model is intended, a one-line comment is enough — `03_Evaluating_your_Judge.ipynb` already makes the case for why it matters.
**How I found them**
By classifying each file by which model *generates* and which *grades*, not by variable name. A grep for `JUDGE_MODEL` matches exactly the files that do it right and misses every one above.
Contributor guide
Research direction
Read commit 4a72984, then compare the seven named notebooks and model_config.py with 02-quality-metrics/03_Evaluating_your_Judge.ipynb and the nine files using separate model constants. Trace each subject and judge entry point, including utils.py:185 and the cited notebook cells. Done means each shared model is separated or explicitly documented as intentional, with the prose and evaluator wiring matching the actual roles.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook
- Domain
- ai, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100