aws / aws/sagemaker-pytorch-inference-toolkit
JVM detect the CPU count as 1 when more CPUs are available for the container.
- Dominant language
- Python
- Stars
- 143
- Forks
- 73
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
This issue is related to the issue [JVM bug 82 in sagemaker-inference-toolkit](https://github.com/aws/sagemaker-inference-toolkit/issues/82)
**To reproduce**
Clone the SaeMaker example
Deploy the model using the same endpoint.
Check CloudWatch logs and the number of CPU cores detected will be like Number of CPUs: 1
JVM detect the CPU count as 1 when more CPUs are available for the container.
**Expected behavior**
The CPU count from CloudWatch should match the CPU count for the used instance. For example, 4 if the instance is ml.m4.xlarge
**System information**
Container: pytorch-inference:1.7-cpu-py3 and pytorch-inference:1.7-gpu-py3
SageMaker inference v1.1.2
**Additional context**
This clearly does not allow the usage of all CPUs on the instance for Sagemaker Inference
Contributor guide
Research direction
Start by deploying the referenced SageMaker example with pytorch-inference:1.7-cpu-py3 and pytorch-inference:1.7-gpu-py3, then inspect CloudWatch's reported CPU count. Compare it with the instance's available CPUs and investigate the JVM/container CPU-detection path; done means the log reports the CPUs available on the instance and inference can use them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100