aws-samples / aws-samples/foundation-model-benchmarking-tool

Add support for different payload formats for bring your own datasets for that might be needed for different inference containers

Open
#5 1 comment 0 reactions 1 assignee Claimed by @madhurprash View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
256
Forks
44
PR merge metrics
No merged PRs in 30d

Description

This tool currently supports the HF TGI container, and DJL Deep Speed container on SageMaker and both use the same format but in future other containers might need a different payload format.

Goal: To give user full flexibility to bring their payloads or contain code that generalizes payload generation irrespective of the container type that the user uses. Two options for solution to this issue here:

1/ Have the user bring in their own payload
2/ Have a generic function defined to convert the payload in support for the container type the user is using to deploy their model and generate inference from.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.