aws / aws/amazon-sagemaker-examples

[Bug Report] sagemaker_batch_inference_torchserve batch is 1 not 3

Open
#3,425 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
Add the link to the notebook.
https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-python-sdk/pytorch_batch_inference/sagemaker_batch_inference_torchserve.ipynb

**Describe the bug**
A clear and concise description of what the bug is.
Run the notebook, i found the batchSize is 1 in cw log, not 3 in the model config.

**To reproduce**
A clear, step-by-step set of instructions to reproduce the bug.

**Logs**
```
Model config:
{
"model": {
"1.0": {
"defaultVersion": true,
"marName": "model.mar",
"minWorkers": 1,
"maxWorkers": 1,
"batchSize": 3,
"maxBatchDelay": 100000,
"responseTimeout": 60
}
}
}
.......
2022-05-22 23:33:22,590 [INFO ] W-9000-model_1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2022-05-22 23:33:22,590 [INFO ] W-9000-model_1-stdout MODEL_LOG - [PID]125
2022-05-22 23:33:22,590 [INFO ] W-9000-model_1-stdout MODEL_LOG - Torch worker started.
2022-05-22 23:33:22,590 [INFO ] W-9000-model_1-stdout MODEL_LOG - Python runtime: 3.8.10
2022-05-22 23:33:22,595 [INFO ] W-9000-model_1 org.pytorch.serve.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.ts.sock.9000
2022-05-22 23:33:22,604 [INFO ] W-9000-model_1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2022-05-22 23:33:22,637 [INFO ] W-9000-model_1-stdout MODEL_LOG - model_name: model, batchSize: 1
```
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.

Contributor guide

Open the contributing guide

Research direction

Start with sagemaker-python-sdk/pytorch_batch_inference/sagemaker_batch_inference_torchserve.ipynb and run the notebook to reproduce the CloudWatch log showing batchSize 1 despite the model config specifying 3. Trace where the TorchServe model configuration is created and deployed, then confirm the resulting worker log reports the configured batch size.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.