aws-samples/foundation-model-benchmarking-tool

View on GitHub

Foundation model benchmarking tool. Run any model on any AWS platform and benchmark for performance across instance type and serving stack options.

Stars: 256
Forks: 44
Open beginner issues: 0
Indexed issues: 36

Dominant language: Jupyter Notebook
License: MIT-0
Last GitHub push: Apr 11, 2025
Latest indexed: Sep 16, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: good first issue help wanted
PR merge metrics: No merged PRs in 30d

Open All

36 open issues indexed

Add support for different payload formats for bring your own datasets for that might be needed for different inference containers Open

enhancement

aws-samples/foundation-model-benchmarking-tool#5 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Emit live metrics Open

help wanted

aws-samples/foundation-model-benchmarking-tool#6 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Add Autoscaling functionality to FMBench Open

enhancement

aws-samples/foundation-model-benchmarking-tool#10 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Code cleanup needed to replace the notebooks with regular python files Open

help wanted

aws-samples/foundation-model-benchmarking-tool#20 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Add documentation for the config file on readthedocs.io Open

good first issue

aws-samples/foundation-model-benchmarking-tool#21 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Add support for models hosted on platforms other than SageMaker Open

enhancement

aws-samples/foundation-model-benchmarking-tool#22 · 0 comments · 2 reactions · 0 assignees · Indexed Sep 16, 2026
The README gives an impression that this tool is comparing instance types when it should be serving stacks Open

documentation

aws-samples/foundation-model-benchmarking-tool#25 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
The business summary plot in the report needs to have a caption for disclaimer Open

bug

aws-samples/foundation-model-benchmarking-tool#26 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Allow different experiments in the same config file to use different regions Open

good first issue

aws-samples/foundation-model-benchmarking-tool#27 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Provide config file for FLAN-T5 out of the box Open

enhancement

aws-samples/foundation-model-benchmarking-tool#28 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Try FMBench with instance count set to > 1 to see how scaling impacts latency and transactions per minute Open

enhancement

aws-samples/foundation-model-benchmarking-tool#29 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Compare different models for the same dataset Open

enhancement

aws-samples/foundation-model-benchmarking-tool#30 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Need for Integration of FMBench Tool with External 3rd party Synthetic Data Generation Tools for Benchmarking usecase. Open

aws-samples/foundation-model-benchmarking-tool#32 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
FMBench to support benchmarking for embedding models Open

enhancement

aws-samples/foundation-model-benchmarking-tool#33 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Loosen dependencies for more flexible installation Open

aws-samples/foundation-model-benchmarking-tool#105 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Reports only show information per instance instead of per endpoint Open

aws-samples/foundation-model-benchmarking-tool#114 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Is it possible to run FMBench on an EC2 instance on endpoints currently not supported on Sagemaker. Open

aws-samples/foundation-model-benchmarking-tool#120 · 5 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Remove the config_filepath.txt as a config reference for developer workflow Open

aws-samples/foundation-model-benchmarking-tool#150 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Unable to locate credential issue Open

aws-samples/foundation-model-benchmarking-tool#153 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 16, 2026
TypeError: unsupported format string passed to NoneType.__format__ Open

aws-samples/foundation-model-benchmarking-tool#154 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
UI Support for FMBench Open

aws-samples/foundation-model-benchmarking-tool#155 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Add retry handling for throttling errors during inference Open

aws-samples/foundation-model-benchmarking-tool#157 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
add hallucination metric to model evals on FMBench Open

aws-samples/foundation-model-benchmarking-tool#170 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Add support for non AWS models to be benchmarked Open

aws-samples/foundation-model-benchmarking-tool#203 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
miss warm up phase Open

aws-samples/foundation-model-benchmarking-tool#213 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Add support for Bedrock Custom Import Models Open

aws-samples/foundation-model-benchmarking-tool#224 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Allow backup/ failover regions to be specified Open

aws-samples/foundation-model-benchmarking-tool#232 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Fails when testing Haiku on AWS Bedrock Open

aws-samples/foundation-model-benchmarking-tool#247 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
parameterize configuration files Open

aws-samples/foundation-model-benchmarking-tool#261 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Experiment config file's model name doesn't match with pricing_fallback.yml file's model name Open

aws-samples/foundation-model-benchmarking-tool#267 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
The pricing fetching logic only works for EC2 Open

aws-samples/foundation-model-benchmarking-tool#268 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Add benchmarking for training models Open

aws-samples/foundation-model-benchmarking-tool#270 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Model accuracy result file doesn't show correct candidate model name Open

aws-samples/foundation-model-benchmarking-tool#276 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Dataset handling if any of the keys have missing fields Open

aws-samples/foundation-model-benchmarking-tool#281 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026
Bring your own judge functionality Open

aws-samples/foundation-model-benchmarking-tool#283 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 16, 2026
Support speculative decoding Open

aws-samples/foundation-model-benchmarking-tool#289 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 16, 2026

aws-samples/foundation-model-benchmarking-tool

Get new issues in your inbox