aws / aws/amazon-sagemaker-examples
[Bug Report] sagemaker_batch_inference_torchserve batch is 1 not 3
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
Add the link to the notebook.
https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-python-sdk/pytorch_batch_inference/sagemaker_batch_inference_torchserve.ipynb
**Describe the bug**
A clear and concise description of what the bug is.
Run the notebook, i found the batchSize is 1 in cw log, not 3 in the model config.
**To reproduce**
A clear, step-by-step set of instructions to reproduce the bug.
**Logs**
```
Model config:
{
"model": {
"1.0": {
"defaultVersion": true,
"marName": "model.mar",
"minWorkers": 1,
"maxWorkers": 1,
"batchSize": 3,
"maxBatchDelay": 100000,
"responseTimeout": 60
}
}
}
.......
2022-05-22 23:33:22,590 [INFO ] W-9000-model_1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2022-05-22 23:33:22,590 [INFO ] W-9000-model_1-stdout MODEL_LOG - [PID]125
2022-05-22 23:33:22,590 [INFO ] W-9000-model_1-stdout MODEL_LOG - Torch worker started.
2022-05-22 23:33:22,590 [INFO ] W-9000-model_1-stdout MODEL_LOG - Python runtime: 3.8.10
2022-05-22 23:33:22,595 [INFO ] W-9000-model_1 org.pytorch.serve.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.ts.sock.9000
2022-05-22 23:33:22,604 [INFO ] W-9000-model_1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2022-05-22 23:33:22,637 [INFO ] W-9000-model_1-stdout MODEL_LOG - model_name: model, batchSize: 1
```
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.
Contributor guide
Research direction
Start with sagemaker-python-sdk/pytorch_batch_inference/sagemaker_batch_inference_torchserve.ipynb and run the notebook to reproduce the CloudWatch log showing batchSize 1 despite the model config specifying 3. Trace where the TorchServe model configuration is created and deployed, then confirm the resulting worker log reports the configured batch size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100