How to dynamically batch to help handle high load?

Hi there,

I'm trying to deploy an endpoint that has bursts of high load. I'd like the endpoint to batch requests so we can increase through put under high load at the cost of a slight increase in latency under low load.

I found a blog post about how this can be done through torchserve and aws. See the section [TorchServe dynamic batching on SageMaker](https://aws.amazon.com/blogs/machine-learning/optimize-your-inference-jobs-using-dynamic-batch-inference-with-torchserve-on-amazon-sagemaker/).

I'd like to have dynamic batching in a huggingface container, as I'm told there are optimizations taken for transformer models there.

I can see the param for `batch_size` in the [handler_service.py](https://github.com/aws/sagemaker-huggingface-inference-toolkit/blob/2f1fae5cbb3b68299e73cc591c0a912b7cccee29/src/sagemaker_huggingface_inference_toolkit/handler_service.py#L52) code, but I'm not sure of the recommended way to adjust this, along with a parameter for `max_batch_delay`.

Is this something currently available?

I reached out to AWS for support, who suggested I open an issue here for assistance. Do let me know if this is more appropriate for a Q&A forum and point me there.

Thanks so much in advance,
Jamie

Start with handler_service.py and the linked TorchServe dynamic-batching post; inspect how batch_size is exposed and whether max_batch_delay is represented. Done means providing an explicit answer on current availability and, if unsupported, defining the implementation scope or documentation path.

Written by the indexing model from the issue text.