[Feature] EvalAlgorithmInterface.evaluate should accept a list of DataConfigs for consistency
- Dominant language
- Python
- Stars
- 291
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
Today [EvalAlgorithmInterface.evaluate](https://github.com/aws/fmeval/blob/220fdc090ee4f1db7174f6e7661a282a5987d1af/src/fmeval/eval_algorithms/eval_algorithm.py#L33) is typed to return `List[EvalOutput]` ("for dataset(s)", per the docstring), but its `dataset_config` argument only accepts `Optional[DataConfig]`.
It seems like most concrete eval algorithms (like [QAAccuracy here](https://github.com/aws/fmeval/blob/220fdc090ee4f1db7174f6e7661a282a5987d1af/src/fmeval/eval_algorithms/qa_accuracy.py#L189)) **either** take the user's `data_config` for a single dataset, **or** take *all* the [pre-defined DATASET_CONFIGS](https://github.com/aws/fmeval/blob/220fdc090ee4f1db7174f6e7661a282a5987d1af/src/fmeval/eval_algorithms/__init__.py#L242) relevant to the evaluator's problem type.
...So the internal logic of evaluators is set up to support providing multiple datasets and returning multiple results already, but we seem to prevent users from calling `evaluate()` with multiple of their own datasets for no particular reason?
Contributor guide
Research direction
Start with src/fmeval/eval_algorithms/eval_algorithm.py and compare its evaluate signature and docstring with a concrete implementation such as src/fmeval/eval_algorithms/qa_accuracy.py. Trace how DATASET_CONFIGS in src/fmeval/eval_algorithms/__init__.py are consumed, then verify that user-provided lists of DataConfigs are accepted and produce the corresponding EvalOutput results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100