aws-samples / aws-samples/foundation-model-benchmarking-tool

Bring your own judge functionality

Open
#283 0 comments 0 reactions 1 assignee Claimed by @madhurprash View on GitHub
Dominant language
Jupyter Notebook
Stars
256
Forks
44
PR merge metrics
No merged PRs in 30d

Description

The current FMBench implementation for evaluations uses judges on Amazon Bedrock via litellm. To add a bring your own judge functionality, we will have to change this to have a base evaluatorClass that will make predictions and calculate the cost (similar to the FMBenchPredictor base class).

Using this implementation, customers will be able to evaluate models using their own judge LLMs in a custom/personalized manner.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.