aws-samples / aws-samples/foundation-model-benchmarking-tool
Need for Integration of FMBench Tool with External 3rd party Synthetic Data Generation Tools for Benchmarking usecase.
Open
- Dominant language
- Jupyter Notebook
- Stars
- 256
- Forks
- 44
- PR merge metrics
- No merged PRs in 30d
Description
1) It would be good to have a native integration in the Config.Yaml file of FMBench for integrating with other 3rd party tools for pulling synthetically generated datasets.
2) How to split this synthetically generated datasets for FMBench based evaluation of FM's would be important functionality to have.
3) Final Results (results.md) generated by this tool could have some Visual comparison capability with other 3rd party tools which could be used for holistic evaluation of FM's.
Contributor guide
Assessment
This issue has not been assessed yet.