aws / aws/sagemaker-pytorch-inference-toolkit

Specify batch size for MME

Open
#134 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
143
Forks
73
PR merge metrics
No merged PRs in 30d

Description

**What did you find confusing? Please describe.**
How do you specify batch size for MME models?

**Describe how documentation can be improved**
[This blog](https://aws.amazon.com/blogs/machine-learning/optimize-your-inference-jobs-using-dynamic-batch-inference-with-torchserve-on-amazon-sagemaker/) describes using env vars to set batch size and other parameters for a single-model endpoint, however, I haven't found any documentation on setting batch size for individual models within a MME.

**Additional context**
Each model in my MME has a `MAR-INF/MANIFEST.json` within its `model.tar.gz`, so I tried to specify `batchSize` in these files, but I don't think it's being applied.

Contributor guide

Open the contributing guide

Research direction

Start with the linked dynamic-batching blog and the model.tar.gz contents, especially MAR-INF/MANIFEST.json. Determine whether batchSize is supported per model in a multi-model endpoint and document the correct configuration or limitation, including what users should expect when the manifest setting is ignored.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.