aws / aws/fmeval

[Feature] EvalAlgorithmInterface.evaluate should accept a list of DataConfigs for consistency

Open
#269 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
291
Forks
60
PR merge metrics
No merged PRs in 30d

Description

Today [EvalAlgorithmInterface.evaluate](https://github.com/aws/fmeval/blob/220fdc090ee4f1db7174f6e7661a282a5987d1af/src/fmeval/eval_algorithms/eval_algorithm.py#L33) is typed to return `List[EvalOutput]` ("for dataset(s)", per the docstring), but its `dataset_config` argument only accepts `Optional[DataConfig]`.

It seems like most concrete eval algorithms (like [QAAccuracy here](https://github.com/aws/fmeval/blob/220fdc090ee4f1db7174f6e7661a282a5987d1af/src/fmeval/eval_algorithms/qa_accuracy.py#L189)) **either** take the user's `data_config` for a single dataset, **or** take *all* the [pre-defined DATASET_CONFIGS](https://github.com/aws/fmeval/blob/220fdc090ee4f1db7174f6e7661a282a5987d1af/src/fmeval/eval_algorithms/__init__.py#L242) relevant to the evaluator's problem type.

...So the internal logic of evaluators is set up to support providing multiple datasets and returning multiple results already, but we seem to prevent users from calling `evaluate()` with multiple of their own datasets for no particular reason?

Contributor guide

Open the contributing guide

Research direction

Start with src/fmeval/eval_algorithms/eval_algorithm.py and compare its evaluate signature and docstring with a concrete implementation such as src/fmeval/eval_algorithms/qa_accuracy.py. Trace how DATASET_CONFIGS in src/fmeval/eval_algorithms/__init__.py are consumed, then verify that user-provided lists of DataConfigs are accepted and produce the corresponding EvalOutput results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.