aws/fmeval

View on GitHub

Foundation Model Evaluations Library

Stars: 291
Forks: 60
Open beginner issues: 0
Indexed issues: 15

Dominant language: Python
License: Apache-2.0
Last GitHub push: Aug 7, 2025
Latest indexed: Sep 14, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed
PR merge metrics: No merged PRs in 30d

Open All

15 open issues indexed

[Feature] LLM-based (QA Accuracy) eval algorithm Open

aws/fmeval#163 · 2 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
[Feature] Support string output path in `EvalAlgorithmInterface.evaluate(save="...")` Open

aws/fmeval#165 · 3 comments · 2 reactions · 0 assignees · Indexed Sep 14, 2026
[Feature] Multi-variable prompt templates Open

aws/fmeval#246 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
[Feature] Image fields for multi-modal models Open

aws/fmeval#247 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
[Feature] JSON export/import for EvalOutput classes Open

aws/fmeval#268 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
[Feature] EvalAlgorithmInterface.evaluate should accept a list of DataConfigs for consistency Open

aws/fmeval#269 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
[Feature] Streaming results/progress and summary metrics for faster feedback Open

aws/fmeval#278 · 1 comment · 2 reactions · 0 assignees · Indexed Sep 14, 2026
[Feature] Add system metrics collected during evaluation to eval_output Open

aws/fmeval#280 · 4 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
[Feature] Add callback mechanism to evaluation Open

aws/fmeval#284 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Option to disable BERTScore in QAAccuracy Open

aws/fmeval#330 · 3 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
[Feature] Lambda model runner Open

aws/fmeval#331 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
[Feature] Increase model coverage of the BERT Score metric by adding torchmetrics implementation Open

aws/fmeval#332 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
Problem running package with python 3.13 Open

aws/fmeval#337 · 2 comments · 2 reactions · 0 assignees · Indexed Sep 14, 2026
Cannot load dataset from s3 Open

aws/fmeval#338 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Community package: fmeval-openeval-adapter — export datasets/results to a portable interchange format Open

aws/fmeval#353 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026

aws/fmeval

Get new issues in your inbox