huggingface / huggingface/evaluate

Issue in evaluate.EvaluationModule.add

Open
#670 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Unlike `evaluate.EvaluationModule.add_batch`, `evaluate.EvaluationModule.add` DOES NOT WORK with Metrics (`evaluate.Metric(evaluate.EvaluationModule)`) whose Features (`evaluate.MetricInfo(evaluate.EvaluationModuleInfo).features`) are `List[datasets.Features]`, instead of `datasets.Features`. Because a `list` DOES NOT HAVE an `encode_example` Method as in Line 533 from `evaluate/src/evaluate/module.py`, throwing a `ValueError` that DOES NOT TRULY RELATE to the cause itself, as it can be seen in:

`ValueError: Predictions and/or references don't match the expected format.
Expected format:
Feature option 0: {'predictions': Value(dtype='string', id='sequence'), 'references': Sequence(feature=Value(dtype='string', id='sequence'), length=-1, id='references')}
Feature option 1: {'predictions': Value(dtype='string', id='sequence'), 'references': Value(dtype='string', id='sequence')},
Input predictions: ['void hello_world()'],
Input references: ['void hello_world()']`

The `evaluate.EvaluationModule.add` Method should instead use the `selected_feature_format` Field from `evaluate.EvaluationModule`, as in Line 486 from `evaluate/src/evaluate/module.py` at `evaluate.EvaluationModule.add_batch`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.