Azure / Azure/azureml-examples
VM Size depending on model
- Dominant language
- Jupyter Notebook
- Stars
- 2k
- Forks
- 1.7k
- Avg merge
- 18h 18m
- Merged PRs (30d)
- 2
Description
### Describe your suggestion
On this notebook:
https://github.com/Azure/azureml-examples/blob/main/sdk/python/foundation-models/system/inference/text-generation/llama-safe-online-deployment.ipynb
You mention:
sku_name = "Standard_NC24s_v3" # Name of the sku(instance type) Check the model-list(can be found in the parent folder(inference)) to get the most optimal sku for your model (Default: Standard_DS2_v2)
There is no model-list file or folder in the inference folder.
Can you please describe how to know which VM size will suffice for a model? I dont have quota for GPUs for example, any other memory optmized would work? how can I know this?
### Additional details
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.