Option to disable BERTScore in QAAccuracy
- Dominant language
- Python
- Stars
- 291
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
The addition of BERTScore to the QAAccuracy in [version 1.2.0](https://github.com/aws/fmeval/releases/tag/v1.2.0) has massively slowed down evaluation: I've seen interactive evaluations (with our custom Streamlit solution [here](https://github.com/aws-samples/llm-evaluation-methodology)) change from ~30 seconds to 3 minutes on upgrading; not to mention some AWS Console created jobs going from less than one hour to ~14 hours or more.
Since this scoring method appears very compute intensive compared to the others, can we have a configuration option in `QAAccuracyConfig` to disable it?
Contributor guide
Research direction
Locate QAAccuracyConfig and the QAAccuracy evaluation path, then read how the scoring methods are selected after the BERTScore addition in version 1.2.0. Check the existing configuration and evaluation tests, if present, before making the option consistent with other scoring settings. Done means callers can disable BERTScore while the remaining QAAccuracy evaluation still runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100