huggingface / huggingface/evaluate
Issue in evaluate.EvaluationModule.add
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
Unlike `evaluate.EvaluationModule.add_batch`, `evaluate.EvaluationModule.add` DOES NOT WORK with Metrics (`evaluate.Metric(evaluate.EvaluationModule)`) whose Features (`evaluate.MetricInfo(evaluate.EvaluationModuleInfo).features`) are `List[datasets.Features]`, instead of `datasets.Features`. Because a `list` DOES NOT HAVE an `encode_example` Method as in Line 533 from `evaluate/src/evaluate/module.py`, throwing a `ValueError` that DOES NOT TRULY RELATE to the cause itself, as it can be seen in:
`ValueError: Predictions and/or references don't match the expected format.
Expected format:
Feature option 0: {'predictions': Value(dtype='string', id='sequence'), 'references': Sequence(feature=Value(dtype='string', id='sequence'), length=-1, id='references')}
Feature option 1: {'predictions': Value(dtype='string', id='sequence'), 'references': Value(dtype='string', id='sequence')},
Input predictions: ['void hello_world()'],
Input references: ['void hello_world()']`
The `evaluate.EvaluationModule.add` Method should instead use the `selected_feature_format` Field from `evaluate.EvaluationModule`, as in Line 486 from `evaluate/src/evaluate/module.py` at `evaluate.EvaluationModule.add_batch`.
Contributor guide
Assessment
This issue has not been assessed yet.