aws / aws/sagemaker-pytorch-inference-toolkit
Specify batch size for MME
- Dominant language
- Python
- Stars
- 143
- Forks
- 73
- PR merge metrics
- No merged PRs in 30d
Description
**What did you find confusing? Please describe.**
How do you specify batch size for MME models?
**Describe how documentation can be improved**
[This blog](https://aws.amazon.com/blogs/machine-learning/optimize-your-inference-jobs-using-dynamic-batch-inference-with-torchserve-on-amazon-sagemaker/) describes using env vars to set batch size and other parameters for a single-model endpoint, however, I haven't found any documentation on setting batch size for individual models within a MME.
**Additional context**
Each model in my MME has a `MAR-INF/MANIFEST.json` within its `model.tar.gz`, so I tried to specify `batchSize` in these files, but I don't think it's being applied.
Contributor guide
Research direction
Start with the linked dynamic-batching blog and the model.tar.gz contents, especially MAR-INF/MANIFEST.json. Determine whether batchSize is supported per model in a multi-model endpoint and document the correct configuration or limitation, including what users should expect when the manifest setting is ignored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100