huggingface / huggingface/Microsoft-Azure

Include inference deployment examples to Azure Kubernetes Service

Open
#30 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Makefile
Stars
16
Forks
6
PR merge metrics
No merged PRs in 30d

Description

The Microsoft Azure Kubernetes Service (AKS) team has curated simple inference deployment examples using the open-source KAITO Sandbox project in the Cloud Native Computing Foundation (CNCF). We propose the following examples using KAITO to deploy Hugging Face LLMs onto an AKS cluster, in a dedicated `aks` directory in `huggingface/Microsoft-Azure/examples` and associated `aks` documentation in `huggingface/Microsoft-Azure/docs/source`. The examples are listed in:

* https://github.com/kaito-project/kaito/tree/main/examples/inference
* BYO models from HuggingFace can easily be deployed with https://github.com/kaito-project/kaito/blob/9e20ccec0b454ab9445ab206871b58c92e0ceae6/examples/custom-model-integration/reference-image-deployment.yaml and specifying the HuggingFace model identifier

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.