huggingface / huggingface/evaluate
CUAD compute_score: undefined AUPR (NaN from np.trapz) is collapsed to 0, the worst attainable score
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
Small one, raised before I write about it in case I have misread it. Line numbers at commit `a7dd338386a4fae9`.
`metrics/cuad/compute_score.py:117-121`: `np.trapz` returns `NaN` when the integral is undefined — for example a degenerate or empty recall array, where there is nothing to integrate. That `NaN` is collapsed to `0`.
In an area-under-precision-recall metric, `0` is not "undefined". It is the **worst attainable score**. So a case where the AUPR could not be computed is recorded as the model achieving the worst possible result, and the two are indistinguishable in any aggregate.
This is CUAD, a legal-contract benchmark, so the aggregate is the number people quote.
## Suggestion
As an option: propagate the `NaN`, or return `None` and exclude the sample, so an uncomputable area is visible as uncomputable rather than as a floor value.
## Disclosure
This appears as one instance in a short methods paper about evaluation instruments that state an invariant and do not apply it — most instances in it are from my own code. I would rather you saw it first, and if it is intended I will record that.
Contributor guide
Research direction
Read metrics/cuad/compute_score.py at lines 117-121, then trace how the AUPR result is used in the CUAD aggregate. Check the behavior for degenerate or empty recall arrays and determine how an undefined result should remain distinguishable from a score of 0; the aggregate should no longer silently treat an uncomputable area as the worst attainable score.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100