feat(judge): support multiple judge types as independent scoring points
- Dominant language
- Go
- Stars
- 894
- Forks
- 68
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 38
Description
## Problem / motivation
A case currently accepts a single `judge`, whose `type` is one of `rule_based`, `script`, or `agent_judge`. This prevents an evaluation from scoring deterministic correctness and semantic quality against the same agent run.
A common example is a code-change evaluation:
- a `script` judge runs tests, validates generated files, or checks structured output;
- an `agent_judge` evaluates whether the change satisfies the user intent, avoids unnecessary complexity, and explains the result well.
These are independent quality dimensions and should be reported as separate scoring points from one execution.
The current alternatives do not fully cover this use case:
- `expect` can coexist with one judge, but only supports built-in checks and acts as a hard pre-check. A failed `expect` skips the judge.
- duplicating the case with different judges runs the agent twice, so the scores may describe different outputs and incur extra execution cost.
- using only `agent_judge` for deterministic checks makes stable assertions more expensive and less reproducible.
This also applies to document generation, data analysis, tool-use workflows, and multi-turn conversations where structural correctness and semantic quality need separate evidence.
## Proposed solution
Support a list of judges at eval or case level while retaining the existing singular `judge` syntax for backward compatibility. For example:
```yaml
judges:
- id: functional-correctness
type: script
script_path: evals/scripts/check-result.sh
timeout_seconds: 30
- id: semantic-quality
type: agent_judge
model: anthropic/claude-sonnet-4-6
criteria:
- "The implementation satisfies the user intent"
- "The change avoids unnecessary complexity"
pass_threshold: 0.5
```
Each configured judge should:
- evaluate the same agent execution and immutable evaluation inputs;
- retain its own timeout, result, evidence, and artifacts;
- be attributable in JSON, JUnit, HTML, Markdown, and grading reports by judge ID and type;
- not silently prevent another scoring point from running unless explicit short-circuit semantics are configured.
The existing `judge` field could be normalized internally to a one-element judge list. Configuring both `judge` and `judges` should produce a validation error.
### Scoring semantics to decide
The initial design should explicitly define whether:
1. each judge is one top-level scoring point, with `agent_judge.criteria` aggregated internally by its existing `pass_threshold`; or
2. all assertions from all judges are flattened into one case-level pass rate.
Treating each judge as one top-level point is likely safer because adding an Agent criterion would otherwise change the relative weight of the script result. Optional fields such as `weight`, `required`, and a case-level scoring threshold could be considered:
```yaml
judges:
- id: functional-correctness
type: script
weight: 0.6
required: true
script_path: evals/scripts/check-result.sh
- id: semantic-quality
type: agent_judge
weight: 0.4
model: anthropic/claude-sonnet-4-6
criteria:
- "The implementation satisfies the user intent"
scoring:
pass_threshold: 0.8
```
Hard safety or correctness gates should remain distinguishable from ordinary weighted score points.
### Execution and isolation considerations
- Script judges currently execute with the case workspace as their working directory and may mutate it. Multiple judges must observe a stable snapshot or run in isolated/read-only contexts so execution order cannot change later scores.
- Judge artifacts should be namespaced, for example `judge//`, rather than sharing a single judge output directory.
- The case timeout and per-judge timeout interaction must be documented so an earlier judge cannot unexpectedly consume the entire budget.
- Multiple judge failures should preserve all available diagnostics rather than only the first failure.
- Eval-level and case-level list inheritance should use documented semantics. A full case-level override is preferable for an initial version over positional array merging.
## Alternatives considered
### Use `expect` plus `agent_judge`
Useful for simple deterministic hard gates, but it cannot execute arbitrary scripts and does not provide two independent judge scores. It also skips Agent judging when the pre-check fails.
### Duplicate the case
This produces two independent agent runs, so the script and Agent scores are not based on the same output. It also increases cost and variability.
### Put deterministic criteria into `agent_judge`
This is more expensive and less reproducible than a script, and it weakens failure attribution.
### Let the Agent judge read script output
This models a dependent pipeline rather than independent scoring points and makes the final semantic result depend on another judge implementation.
## Acceptance criteria
- [ ] A case can configure at least one `script` judge and one `agent_judge` simultaneously.
- [ ] Both judges evaluate the same agent run without rerunning the evaluated agent.
- [ ] Each result is reported with stable judge identity, type, pass/fail status, and evidence.
- [ ] Overall case aggregation and short-circuit behavior are explicitly documented and tested.
- [ ] Judge workspaces and artifacts are isolated so one judge cannot contaminate another.
- [ ] Existing configs using singular `judge` retain their current behavior.
- [ ] Invalid combinations and duplicate judge IDs fail during `skill-up validate`.
- [ ] English and Chinese configuration documentation include a combined example.
- [ ] Unit and report tests cover mixed pass/fail, timeout, error, and artifact cases.
Contributor guide
Research direction
Start by tracing the existing singular judge configuration and the `skill-up validate` entry point; the issue does not name implementation files. Review the current unit and report tests, along with the English and Chinese configuration documentation. Done means mixed judges run against one agent execution, remain isolated, report independently, preserve singular-judge behavior, and cover validation, aggregation, timeout, error, and artifact cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100