huggingface / huggingface/evaluate

CUAD compute_score: undefined AUPR (NaN from np.trapz) is collapsed to 0, the worst attainable score

Open
#801 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Small one, raised before I write about it in case I have misread it. Line numbers at commit `a7dd338386a4fae9`.

`metrics/cuad/compute_score.py:117-121`: `np.trapz` returns `NaN` when the integral is undefined — for example a degenerate or empty recall array, where there is nothing to integrate. That `NaN` is collapsed to `0`.

In an area-under-precision-recall metric, `0` is not "undefined". It is the **worst attainable score**. So a case where the AUPR could not be computed is recorded as the model achieving the worst possible result, and the two are indistinguishable in any aggregate.

This is CUAD, a legal-contract benchmark, so the aggregate is the number people quote.

## Suggestion

As an option: propagate the `NaN`, or return `None` and exclude the sample, so an uncomputable area is visible as uncomputable rather than as a floor value.

## Disclosure

This appears as one instance in a short methods paper about evaluation instruments that state an invariant and do not apply it — most instances in it are from my own code. I would rather you saw it first, and if it is intended I will record that.

Contributor guide

Open the contributing guide

Research direction

Read metrics/cuad/compute_score.py at lines 117-121, then trace how the AUPR result is used in the CUAD aggregate. Check the behavior for degenerate or empty recall arrays and determine how an undefined result should remain distinguishable from a score of 0; the aggregate should no longer silently treat an uncomputable area as the worst attainable score.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.