Azure / Azure/azureml-examples
Azure ML deployment: setting low memory request not taking effect
- Dominant language
- Jupyter Notebook
- Stars
- 2k
- Forks
- 1.7k
- Avg merge
- 18h 18m
- Merged PRs (30d)
- 2
Description
### Operating System
Windows
### Version Information
OS: Azure DevOps windows-latest agent (Windows Server 2022 with Visual Studio 2022)
azure-cli 2.51.0
core 2.51.0
telemetry 1.1.0
Extensions:
azure-devops 0.26.0
ml 2.19.1
Dependencies:
msal 1.24.0b1
azure-mgmt-resource 23.1.0b2
### Steps to reproduce
In Azure Machine Learning, there is an inference cluster named "reco-inference" which is a Azure Kubernetes cluster.
There is a custom instance type named "smallmemoryinstancetype" with 100Mi memory request, created like this:
```kubectl
kubectl apply -f smallmemory_instancetype.yaml
```
where the `smallmemory_instancetype.yaml` file contains:
```yaml
apiVersion: amlarc.azureml.com/v1alpha1
kind: InstanceType
metadata:
name: smallmemoryinstancetype
spec:
resources:
limits:
cpu: "1"
memory: "2Gi"
requests:
cpu: "10m"
memory: "100Mi"
```
There is an azure ML environment: "machine-learning-recommendation-environment:12" (Linux, python version: 3.8).
There is a previously registered model: name: "modelname", version: 1. (Model artifact binary size: 78 mb.)
There is an endpoint named "endpointname" created like this:
```ps
az ml online-endpoint create --name endpointname --set compute=azureml:reco-inference
```
We deploy like this:
```ps
$azuremlModelId = "azureml:modelname:1"
az ml online-deployment create --name deploymentname -f deploymentConfigTest.yaml --set endpoint_name=endpointname --set model=$azuremlModelId --set environment=azureml:machine-learning-recommendation-environment:12
```
where `deploymentConfigTest.yaml` is:
```yaml
type: kubernetes
app_insights_enabled: true
code_configuration:
code: .
scoring_script: score.py
request_settings:
request_timeout_ms: 3000
max_queue_wait_ms: 3000
instance_type: smallmemoryinstancetype
instance_count: 1
scale_settings:
type: default
```
The deployment is successful. It appears in `kubectl describe node` as a pod. I can verify that the instance type is successfully set for the deployment (at the endpoint in Azure Machine Learning Studio).

### Expected behavior
When inspected with `kubectl describe node`, I can see the Memory Requests for the pod of my deployment.
I would expect it to be set to a smaller amount than 500Mi.
### Actual behavior
When inspected with `kubectl describe node`, I can see the Memory Requests for the pod of my deployment.
It is exactly 500Mi.
```
Non-terminated Pods: (13 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
default deploymentname-endpointname-54d8bf5d5w9dz 110m (5%) 1100m (57%) 500Mi (10%) 2098Mi (45%) 18h
...
```
(I also verified that if I use an instance type with more than 500Mi memory request then more than 500Mi will be used, so the instance type setting itself is taking an effect on memory requests.)
As I understand, this is because of the storageinitializer init container (that is in the same pod as my inference server), where memory request is a fixed 500Mi amount.
For example, in the pod settings inspected in Lens, I see these settings for the init container:
```yaml
initContainers:
- name: storageinitializer-modeldata
...
resources:
limits:
cpu: 100m
memory: 500Mi
requests:
cpu: 100m
memory: 500Mi
...
```
### Addition information
My question is: why is the memory request I set on my deployment not taking an effect on the init container also?
Is there any other (maybe completely different) solution to achieve **smaller memory request for the init container**? (For example, it would be ideal if I could set the request size for the init container also dynamically when running the `online-deployment create` command.)
The reason for my question: we would like to deploy several small deployments, but it is very wasteful to use 500 mb memory for each of them (when eg. 65Mi would be sufficient).
(Or is it possible that the init container actually needs this much space to work and I should not try to set the memory request?)
Thank you in advance for your help!
Contributor guide
Research direction
Start by reproducing the deployment from deploymentConfigTest.yaml and smallmemory_instancetype.yaml, then inspect the resulting pod with kubectl describe node and its storageinitializer-modeldata init container. Trace the online-deployment create configuration to determine whether init-container resources are configurable; done means documenting a supported way to reduce the request or confirming that the fixed 500Mi behavior is expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, kubernetes, python
- Domain
- cloud, infrastructure, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100