huggingface / huggingface/evaluate
Implement `get_evaluation_module_config_names()` function
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
In the `datasets` library, there is a handy `get_dataset_config_names()` function ([docs](https://huggingface.co/docs/datasets/v2.2.1/en/package_reference/loading_methods#datasets.get_dataset_config_names)) that returns all the available config names for a given dataset.
It would be useful to have a similar function in `evaluate`, since metrics like `glue` and `bleurt` have multiple config names, and programatic access to these options is handy.
However, a quick look at the `evaluate` codebase shows this isn't quite as easy to achieve as in `datasets`. The main limitation seems to be the absence of a builder config for metrics / evaluation modules. This is used in `datasets` [here](https://github.com/huggingface/datasets/blob/747f0c8612d4929dbdd1c72cca201815911ce660/src/datasets/inspect.py#L224) to quickly inspect the set of possible config names.
cc @TristanThrush this would allow us to avoid needing to import every metric explicitly in the evaluation service UI (when we check for supported metrics).
Contributor guide
Assessment
This issue has not been assessed yet.