huggingface / huggingface/evaluate
SQuAD v2 is missing the NoAns metrics
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
The section "Output values" in the [documentation](https://github.com/huggingface/evaluate/blob/main/metrics/squad_v2/README.md) for the SQuAD v2 metric shows that there should be 13 different measures returned by `metric.compute( )`, yet, as is shown in the examples in the section "Examples", all the measures about "NoAns" are missing in the output.
This is not just a documentation error; running the examples in `evaluate` 0.3.0 gives the same result, without NoAns.
On the topic of documentation (although this is a separate issue): `exact`, `HasAns_exact` and `NoAns_exact` are described with the same sentence, but they clearly should have different meanings. Furthermore, I don't see what the difference is between `NoAns_exact` and `NoAns_f1`, because if there is no answer, then the reference span is empty and hence F1 is measured over a reference of 0 words.
Contributor guide
Assessment
This issue has not been assessed yet.