aws / aws/sagemaker-huggingface-inference-toolkit

Support load_lora_weights in inference API deploy

Open
#131 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
270
Forks
60
PR merge metrics
No merged PRs in 30d

Description

Currently there is no way to add `load_lora_weights` in deployment

```
hub = {
'HF_MODEL_ID': 'black-forest-labs/FLUX.1-dev',
'HF_TASK':'text-to-image',
'HF_TOKEN':'TOKEN'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
env=hub, # configuration for loading model from Hub
role=role, # IAM role with permissions to create an endpoint
transformers_version="4.26", # Transformers version used
pytorch_version="1.13", # PyTorch version used
py_version='py39', # Python version used
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
initial_instance_count=1,
instance_type="ml.g4dn.xlarge"
)
```

Maybe in hub, there could be a new env var as "HF_LORA_MODEL"

Similar implementation present in here https://github.com/aws-samples/sagemaker-stablediffusion-quick-kit/compare/bd37fe958ec8ea27f3e25b7a4c41a3bf9b660bb8...2d1c43be0aec8c20791dd12c2268231c60282f64

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the inference API deployment consumes the hub environment in the repository, then compare the linked sagemaker-stablediffusion-quick-kit implementation. Determine how deployment should expose load_lora_weights, including whether HF_LORA_MODEL is sufficient; done means a deployed model can load LoRA weights through the inference API.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.