Azure / Azure/azureml-examples

VM Size depending on model

Open
#2,523 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
2k
Forks
1.7k
Avg merge
18h 18m
Merged PRs (30d)
2

Description

### Describe your suggestion

On this notebook:
https://github.com/Azure/azureml-examples/blob/main/sdk/python/foundation-models/system/inference/text-generation/llama-safe-online-deployment.ipynb

You mention:
sku_name = "Standard_NC24s_v3" # Name of the sku(instance type) Check the model-list(can be found in the parent folder(inference)) to get the most optimal sku for your model (Default: Standard_DS2_v2)

There is no model-list file or folder in the inference folder.

Can you please describe how to know which VM size will suffice for a model? I dont have quota for GPUs for example, any other memory optmized would work? how can I know this?

### Additional details

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.