Recipe idea: catching LLM-as-judge bias in agent evals with open models

Hi! I'd like to contribute a notebook on evals that quietly lie, with open models from the Hub.

The recipe would take a small agent task and show, with runnable cells, how a single LLM-as-judge score can mislead: position and verbosity bias in the judge, stale test sets, and distribution shift between the eval set and real traffic. Each problem gets a measurable check and a fix, ending with a small multi-signal eval that a reader can reuse.

It builds on my repo https://github.com/TejasQ/basically-agent-evals and my talk "Your Evals Are Lying to You" (accepted at AI Engineer World's Fair 2026). I'm an AI engineer at IBM.

Does this fit the cookbook, and is anyone already working on something similar? If it's welcome I'll open a PR following the contribution guidelines and tag @merveenoyan and @stevhliu.

Start by reading the cookbook contribution guidelines and inspecting the linked basically-agent-evals repository and talk for the proposed approach. Build a runnable notebook around a small agent task, covering position and verbosity bias, stale test sets, and distribution shift, with measurable checks, fixes, and a reusable multi-signal evaluation at the end.

Written by the indexing model from the issue text.