huggingface / huggingface/evaluate

Add `zero_division` argument to F1

Open
#699 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

If you compute F1 in a case where a class is neither predicted nor present in the ground truth data, a warning refers to using `zero_division`:
```python
import evaluate
f1 = evaluate.load("f1")
print(f1.compute(predictions=[0,0,0,0,0], references=[0,1,0,1,2], average=None, labels=[0,1,2,3]))
```

Output:
```
{'f1': array([0.57142857, 0. , 0. , 0. ])}
[...\.venv\Lib\site-packages\sklearn\metrics\_classification.py:1731](file:///.../.venv/Lib/site-packages/sklearn/metrics/_classification.py:1731): UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true nor predicted samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, f"{metric.capitalize()} is", result.shape[0])
```

But this argument is not present:
```python
import evaluate, numpy as np
f1 = evaluate.load("f1")
print(f1.compute(predictions=[0,0,0,0,0], references=[0,1,0,1,2], average=None, labels=[0,1,2,3], zero_division=np.nan))
```

Output:
```
TypeError Traceback (most recent call last)
Cell In[4], [line 3](vscode-notebook-cell:?execution_count=4&line=3)
1 import evaluate, numpy as np
2 f1 = evaluate.load("f1")
----> [3](vscode-notebook-cell:?execution_count=4&line=3) print(f1.compute(predictions=[0,0,0,0,0], references=[0,1,0,1,2], average=None, labels=[0,1,2,3], zero_division=np.nan))

File ...\.venv\Lib\site-packages\evaluate\module.py:467, in EvaluationModule.compute(self, predictions, references, **kwargs)
465 inputs = {input_name: self.data[input_name][:] for input_name in self._feature_names()}
466 with temp_seed(self.seed):
--> [467](file:///.../.venv/Lib/site-packages/evaluate/module.py:467) output = self._compute(**inputs, **compute_kwargs)
469 if self.buf_writer is not None:
470 self.buf_writer = None

TypeError: F1._compute() got an unexpected keyword argument 'zero_division'
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.