PaperFinder scorer: criterion-name matching is case-sensitive, silently dropping documents from relevance scoring

# PaperFinder scorer: criterion-name matching is case-sensitive, silently dropping documents from relevance scoring

**Version:** `asta-bench` v0.5.4 (`8fbdbbb`)
**Affects:** `astabench/evals/paper_finder/relevance.py`, all `semantic_*` (broad) queries

## Summary

`_build_relevance_judgement_results` verifies that the LLM judge returned a judgement for
every required criterion, and does so with a **case-sensitive** string comparison
(`relevance.py:226-228`):

```python
required_criteria_names = [criterion.name for criterion in relevance_criteria]
judgement_criteria_names = [criterion["name"] for criterion in rcj_result]
if any(name not in judgement_criteria_names for name in required_criteria_names):
logger.warning("Required relevance criteria not found ... Skipping ...")
continue
```

`gpt-4o-2024-11-20` is asked for each name "exactly as given in the provided criteria"
and usually complies, but not always — it frequently lowercases acronyms. Observed
repeatedly on `semantic_8`:

```
WARNING sample=semantic_8 relevance.py:229
Required relevance criteria not found for document 227162720. Skipping.
Required criteria: ['decoupled workers', 'distributed RL'].
Judged criteria: ['decoupled workers', 'distributed rl'].
```

The judgement is correct and complete. Only the capitalisation of `RL` differs, and the
document is dropped.

## Why this looks like an oversight rather than a design choice

The line immediately above already normalises the *other* common LLM naming error, with
a comment saying so (`relevance.py:195-196`):

```python
# NOTE: removing underscore as it's a common llm error for naming fields
"name": criterion_name.replace("_", " "),
```

Case drift is the same class of error from the same model, and is not handled.

## Impact

**1. Documents are dropped, not scored zero.** The `continue` means the paper never
enters `judgements`, so it is absent from both the `rank` term
(`lower_bound_corrected_ndcg` over `judgements.values()`) and the recall numerator in
`calc_recall_at_k`. A paper the judge rated `perfectly_relevant` is simply lost.

**2. The exposure is nearly total.** In `paper_finder_validation`, across the 48 queries
that carry `relevance_criteria`, **138 of 142 criterion names (97.2%) contain at least
one uppercase character**, so almost every criterion can trigger this. Whether it fires
is per-judge-call nondeterminism.

**3. Runs are not reproducible, and the judge is re-billed every time.** Skipped
documents never reach the judgement cache. `get_llm_relevance` (`eval.py:51-83`) reuses
`detailed_reference[qid][paper_id]` when present and otherwise judges, then writes back
only `new_judgements` — which never contains a skipped document. So those papers are
re-judged on every run and get a fresh coin flip each time.

We observed this directly: two runs of the **same 8 samples with byte-identical
submissions** scored `semantic_f1` 0.1280 and 0.1344, with `New judgements: 0` reported
for cache-hit samples. The re-diced, never-cached documents are the only moving part.

## Proposed fix

Normalise both sides of both comparisons. (The second site matters: fixing only the
membership check moves the failure rather than removing it, because
`calculate_relevance_criteria_score` looks the weight up with a bare `[]` at
`relevance.py:269` and raises `KeyError`, which the caller at `relevance.py:238-242`
turns back into a skipped document.)

```python
def _normalize_criterion_name(name: str) -> str:
return name.replace("_", " ").strip().lower()
```

```diff
@@ _build_relevance_judgement_results
required_criteria_names = [criterion.name for criterion in relevance_criteria]
judgement_criteria_names = [criterion["name"] for criterion in rcj_result]
+ judged = {_normalize_criterion_name(n) for n in judgement_criteria_names}
if any(
- name not in judgement_criteria_names for name in required_criteria_names
+ _normalize_criterion_name(name) not in judged
+ for name in required_criteria_names
):

@@ calculate_relevance_criteria_score
for criteria in relevance_criteria:
- criterion_name_to_weight[criteria.name] = criteria.weight
+ criterion_name_to_weight[_normalize_criterion_name(criteria.name)] = criteria.weight

score = 0
for judgement in judgements:
- # relevance is 0-3; divide by 3 to normalize to a 0-1 score
- score += (
- criterion_name_to_weight[judgement["name"]] * judgement["relevance"] / 3
- )
+ weight = criterion_name_to_weight.get(
+ _normalize_criterion_name(judgement["name"])
+ )
+ if weight is None:
+ continue
+ # relevance is 0-3; divide by 3 to normalize to a 0-1 score
+ score += weight * judgement["relevance"] / 3
```

## Verification

Exercised against `semantic_8`'s real criteria and weights
(`decoupled workers` 0.5, `distributed RL` 0.5):

| judge output | before | after |
|---|---|---|
| `['decoupled workers', 'distributed rl']` (the observed failure) | **SKIPPED** | scored 3 |
| `['decoupled workers', 'distributed RL']` (exact, as instructed) | scored 3 | scored 3 |
| `['decoupled workers']` (genuinely missing a criterion) | SKIPPED | **SKIPPED** |

The last row is the important one: the check still does its job. The fix makes the
comparison insensitive to case, not permissive about missing judgements.

**Name collisions are not a concern on this dataset:** of the 142 criterion names across
the 48 validation queries, lowercasing produces **zero** collisions within any query, so
no information is lost by collapsing case.

## Alternative worth considering

Since the judge is explicitly instructed to echo the names, a stricter alternative is to
ignore the returned keys entirely and match criteria positionally, or to have the judge
return a list indexed by the input order. That is a larger change; the normalisation
above is the minimal fix and is consistent with the existing underscore handling.

## Environment

- `asta-bench` v0.5.4, commit `8fbdbbb`
- `GRADER_MODEL_NAME = "openai/gpt-4o-2024-11-20"` (`relevance.py:21`)
- Task: `astabench/paper_finder_validation`, `semantic_*` samples