aws-samples / aws-samples/foundation-model-benchmarking-tool
Bring your own judge functionality
Open
- Dominant language
- Jupyter Notebook
- Stars
- 256
- Forks
- 44
- PR merge metrics
- No merged PRs in 30d
Description
The current FMBench implementation for evaluations uses judges on Amazon Bedrock via litellm. To add a bring your own judge functionality, we will have to change this to have a base evaluatorClass that will make predictions and calculate the cost (similar to the FMBenchPredictor base class).
Using this implementation, customers will be able to evaluate models using their own judge LLMs in a custom/personalized manner.
Contributor guide
Assessment
This issue has not been assessed yet.