huggingface / huggingface/evaluate

Implement `get_evaluation_module_config_names()` function

Open
#138 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

In the `datasets` library, there is a handy `get_dataset_config_names()` function ([docs](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/loading_methods#datasets.get_dataset_config_names)) that returns all the available config names for a given dataset.

It would be useful to have a similar function in `evaluate`, since metrics like `glue` and `bleurt` have multiple config names, and programatic access to these options is handy.

However, a quick look at the `evaluate` codebase shows this isn't quite as easy to achieve as in `datasets`. The main limitation seems to be the absence of a builder config for metrics / evaluation modules. This is used in `datasets` [here](https://github.com/huggingface/datasets/blob/747f0c8612d4929dbdd1c72cca201815911ce660/src/datasets/inspect.py#L224) to quickly inspect the set of possible config names.

cc @TristanThrush this would allow us to avoid needing to import every metric explicitly in the evaluation service UI (when we check for supported metrics).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.