huggingface / huggingface/evaluate
Add `zero_division` argument to F1
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
If you compute F1 in a case where a class is neither predicted nor present in the ground truth data, a warning refers to using `zero_division`:
```python
import evaluate
f1 = evaluate.load("f1")
print(f1.compute(predictions=[0,0,0,0,0], references=[0,1,0,1,2], average=None, labels=[0,1,2,3]))
```
Output:
```
{'f1': array([0.57142857, 0. , 0. , 0. ])}
[...\.venv\Lib\site-packages\sklearn\metrics\_classification.py:1731](file:///.../.venv/Lib/site-packages/sklearn/metrics/_classification.py:1731): UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true nor predicted samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, f"{metric.capitalize()} is", result.shape[0])
```
But this argument is not present:
```python
import evaluate, numpy as np
f1 = evaluate.load("f1")
print(f1.compute(predictions=[0,0,0,0,0], references=[0,1,0,1,2], average=None, labels=[0,1,2,3], zero_division=np.nan))
```
Output:
```
TypeError Traceback (most recent call last)
Cell In[4], [line 3](vscode-notebook-cell:?execution_count=4&line=3)
1 import evaluate, numpy as np
2 f1 = evaluate.load("f1")
----> [3](vscode-notebook-cell:?execution_count=4&line=3) print(f1.compute(predictions=[0,0,0,0,0], references=[0,1,0,1,2], average=None, labels=[0,1,2,3], zero_division=np.nan))
File ...\.venv\Lib\site-packages\evaluate\module.py:467, in EvaluationModule.compute(self, predictions, references, **kwargs)
465 inputs = {input_name: self.data[input_name][:] for input_name in self._feature_names()}
466 with temp_seed(self.seed):
--> [467](file:///.../.venv/Lib/site-packages/evaluate/module.py:467) output = self._compute(**inputs, **compute_kwargs)
469 if self.buf_writer is not None:
470 self.buf_writer = None
TypeError: F1._compute() got an unexpected keyword argument 'zero_division'
```
Contributor guide
Assessment
This issue has not been assessed yet.