aws / aws/sagemaker-pytorch-inference-toolkit

JVM detect the CPU count as 1 when more CPUs are available for the container.

Open
#99 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
143
Forks
73
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
This issue is related to the issue [JVM bug 82 in sagemaker-inference-toolkit](https://github.com/aws/sagemaker-inference-toolkit/issues/82)

**To reproduce**

Clone the SaeMaker example
Deploy the model using the same endpoint.
Check CloudWatch logs and the number of CPU cores detected will be like Number of CPUs: 1
JVM detect the CPU count as 1 when more CPUs are available for the container.

**Expected behavior**
The CPU count from CloudWatch should match the CPU count for the used instance. For example, 4 if the instance is ml.m4.xlarge

**System information**
Container: pytorch-inference:1.7-cpu-py3 and pytorch-inference:1.7-gpu-py3
SageMaker inference v1.1.2

**Additional context**
This clearly does not allow the usage of all CPUs on the instance for Sagemaker Inference

Contributor guide

Open the contributing guide

Research direction

Start by deploying the referenced SageMaker example with pytorch-inference:1.7-cpu-py3 and pytorch-inference:1.7-gpu-py3, then inspect CloudWatch's reported CPU count. Compare it with the instance's available CPUs and investigate the JVM/container CPU-detection path; done means the log reports the CPUs available on the instance and inference can use them.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.