vllm-project / vllm-project/aibrix
Loading base model/lora adapters from s3 fails and throws HF error
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 694
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 98
Description
### 🐛 Describe the bug
Im trying to initialize AIBrix using model files hosted on s3. But I am not able to get vllm-openai container to run as its incorrectly identifying the s3 link as being a HF repo.
`huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 's3://s3-bucket/llama-3.1-nemoguard-8b-topic-control'. Use `repo_type` argument if needed.`
### Steps to Reproduce
1. Move a model `llama-3.1-nemoguard-8b-topic-control` to a s3 bucket
2. Deploy AIBrix following the docs: https://aibrix.readthedocs.io/latest/features/lora-dynamic-loading.html#create-base-model. Be sure to reference the model using its s3 path: `s3://s3-bucket/llama-3.1-nemoguard-8b-topic-control`
3. Inspect the pod logs and find the error. Pod will crash when applying.
4. Same issue if you just load LoRA from s3
### Expected behavior
Pod should just start running like when downloading model from HF
### Environment
- AIBrix version: 0.20.0
- k8s deployment
- AWS s3 bucket with model files
Contributor guide
Research direction
Begin with the lora-dynamic-loading documentation's create-base-model steps and inspect the vllm-openai pod startup path that handles model paths. Reproduce the failure with the listed s3:// model path, then test both base-model and LoRA loading. Done means the pod starts and loads S3-hosted models without raising HFValidationError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100