aws / aws/fmeval

Option to disable BERTScore in QAAccuracy

Open
#330 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
291
Forks
60
PR merge metrics
No merged PRs in 30d

Description

The addition of BERTScore to the QAAccuracy in [version 1.2.0](https://github.com/aws/fmeval/releases/tag/v1.2.0) has massively slowed down evaluation: I've seen interactive evaluations (with our custom Streamlit solution [here](https://github.com/aws-samples/llm-evaluation-methodology)) change from ~30 seconds to 3 minutes on upgrading; not to mention some AWS Console created jobs going from less than one hour to ~14 hours or more.

Since this scoring method appears very compute intensive compared to the others, can we have a configuration option in `QAAccuracyConfig` to disable it?

Contributor guide

Open the contributing guide

Research direction

Locate QAAccuracyConfig and the QAAccuracy evaluation path, then read how the scoring methods are selected after the BERTScore addition in version 1.2.0. Check the existing configuration and evaluation tests, if present, before making the option consistent with other scoring settings. Done means callers can disable BERTScore while the remaining QAAccuracy evaluation still runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.