vllm-project / vllm-project/aibrix

Loading base model/lora adapters from s3 fails and throws HF error

Open
#766 0 comments 0 reactions 0 assignees View on GitHub
area/lora kind/bug
Dominant language
Go
Stars
5.1k
Forks
694
Avg merge
1d 19h
Merged PRs (30d)
98

Description

### 🐛 Describe the bug

Im trying to initialize AIBrix using model files hosted on s3. But I am not able to get vllm-openai container to run as its incorrectly identifying the s3 link as being a HF repo.

`huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 's3://s3-bucket/llama-3.1-nemoguard-8b-topic-control'. Use `repo_type` argument if needed.`

### Steps to Reproduce

1. Move a model `llama-3.1-nemoguard-8b-topic-control` to a s3 bucket
2. Deploy AIBrix following the docs: https://aibrix.readthedocs.io/latest/features/lora-dynamic-loading.html#create-base-model. Be sure to reference the model using its s3 path: `s3://s3-bucket/llama-3.1-nemoguard-8b-topic-control`
3. Inspect the pod logs and find the error. Pod will crash when applying.
4. Same issue if you just load LoRA from s3

### Expected behavior

Pod should just start running like when downloading model from HF

### Environment

- AIBrix version: 0.20.0
- k8s deployment
- AWS s3 bucket with model files

Contributor guide

Open the contributing guide

Research direction

Begin with the lora-dynamic-loading documentation's create-base-model steps and inspect the vllm-openai pod startup path that handles model paths. Reproduce the failure with the listed s3:// model path, then test both base-model and LoRA loading. Done means the pod starts and loads S3-hosted models without raising HFValidationError.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.