aws-samples / aws-samples/foundation-model-benchmarking-tool
Add support for different payload formats for bring your own datasets for that might be needed for different inference containers
- Dominant language
- Jupyter Notebook
- Stars
- 256
- Forks
- 44
- PR merge metrics
- No merged PRs in 30d
Description
This tool currently supports the HF TGI container, and DJL Deep Speed container on SageMaker and both use the same format but in future other containers might need a different payload format.
Goal: To give user full flexibility to bring their payloads or contain code that generalizes payload generation irrespective of the container type that the user uses. Two options for solution to this issue here:
1/ Have the user bring in their own payload
2/ Have a generic function defined to convert the payload in support for the container type the user is using to deploy their model and generate inference from.
Contributor guide
Assessment
This issue has not been assessed yet.