huggingface / huggingface/evaluate

Token classification bootstrap crashing with custom dataset

Open
#546 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

OS: maxOS 14.1
Python: 3.11.6
PyTorch: 2.0.1

Description: Standard evaluation of custom dataset works:

```
dataset = Dataset.from_list(dictlist)

task_evaluator = evaluator("token-classification")

eval_results = task_evaluator.compute(
model_or_pipeline=,
data=,
metric="seqeval",
label_column="tags"
)
```
However, when adding bootstrapping, I get a crash:
```
eval_results = task_evaluator.compute(
model_or_pipeline=,
data=,
metric="seqeval",
label_column="tags",
strategy="bootstrap",
n_resamples=30,
```

Potential solution: Add parameter for "zero_division" as explained in warning below.

Stacktrace:
UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, msg_start, len(result))
UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use `zero_division` parameter to control this behavior.
_warn_prf(average, modifier, msg_start, len(result))
Traceback (most recent call last):
, line 32, in
eval_results = task_evaluator.compute(
^^^^^^^^^^^^^^^^^^^^^^^
line 266, in compute
metric_results = self.compute_metric(
^^^^^^^^^^^^^^^^^^^^
line 531, in compute_metric
bootstrap_dict = self._compute_confidence_interval(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
line 147, in _compute_confidence_interval
bs = bootstrap(
^^^^^^^^^^
line 450, in bootstrap
args = _bootstrap_iv(data, statistic, vectorized, paired, axis,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
line 155, in _bootstrap_iv
sample = np.atleast_1d(sample)
^^^^^^^^^^^^^^^^^^^^^
line 65, in atleast_1d
ary = asanyarray(ary)
^^^^^^^^^^^^^^^
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (416,) + inhomogeneous part.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.