aws / aws/fmeval

[Feature] Increase model coverage of the BERT Score metric by adding torchmetrics implementation

Open
#332 2 comments 0 reactions 1 assignee Claimed by @Riccardo231 View on GitHub
Dominant language
Python
Stars
291
Forks
60
PR merge metrics
No merged PRs in 30d

Description

The current underlying [implementation of BERT score](https://github.com/Tiiiger/bert_score#readme) supports a [limited set of transformer models](https://docs.google.com/spreadsheets/d/1RKOVpselB98Nnh_EOC4A2BYn8_201tmPODpNWu4w7xI/edit?gid=0#gid=0), and FMEval further truncates this list to `microsoft/deberta-xlarge-mnli` and `roberta-large-mnli`.

[Torchmetrics](https://lightning.ai/docs/torchmetrics/stable/gallery/text/bertscore.html) provides a more generic BERT score implementation. The specific request here is to not limit what transformer models users can configure. The behavior should be as follows:

- If `microsoft/deberta-xlarge-mnli` and `roberta-large-mnli` is specified, use the `bert-score` implementation to avoid regressions for existing customers.
- Otherwise, use the `torchmetrics` implementation of BERT score.

Use cases for broader set of models underlying BERT score:

1. Monolingual BERT models have been shown to outperform multi-lingual BERT models on certain tasks. https://aclanthology.org/2021.acl-long.243.pdf
2. Customers may fine tune their own transformers which can be downloaded into the container running AWSFMeval and passed into the `torchmetrics` BERT score implementation.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.