aws-samples / aws-samples/foundation-model-benchmarking-tool

Compare different models for the same dataset

Open
#30 0 comments 0 reactions 1 assignee Claimed by @aarora79 View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
256
Forks
44
PR merge metrics
No merged PRs in 30d

Description

There is nothing in FMBench which prevents different experiments in the same config file to use different models, but the generated report is not architected in the same way i.e. it is not created to compare different models but rather compare the same model across serving stacks, so that would need to change. This has been requested by multiple customers, the idea being if we find different models that are fit for task, we now want to find the model and serving stack combination which provides the best price:performance.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.