vllm-project / vllm-project/aibrix
Use sidecar to download artifact will block the health check and result in restart
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 697
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 104
Description
🐛 Describe the bug
related to #1826, when use the sidecar to download the artifact from the s3 like storage it will spend some time, for example:
2026-01-14 23:03:15,548 - app.py:318 - main - INFO - {"positional_args": ["Namespace(host=None, port=8080, enable_fastapi_docs=False)"], "event": "Use %s to startup runtime server", "logger": "aibrix.app", "level": "info", "timestamp": "2026-01-14 23:03:15 UTC"}
2026-01-14 23:03:15,548 - app.py:91 - mount_metrics - INFO - {"event": "AIBrix to use /tmp/aibrix/metrics/ as PROMETHEUS_MULTIPROC_DIR", "logger": "aibrix.app", "level": "info", "timestamp": "2026-01-14 23:03:15 UTC"}
2026-01-14 23:03:15,549 - app.py:103 - mount_metrics - INFO - {"event": "AIBrix to scrape metrics from http://localhost:8000/metrics, use vllm standard rules", "logger": "aibrix.app", "level": "info", "timestamp": "2026-01-14 23:03:15 UTC"}
INFO: Started server process [1]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO: 10.0.151.187:53470 - "GET /healthz HTTP/1.1" 200 OK
INFO: 10.0.151.187:53486 - "GET /healthz HTTP/1.1" 200 OK
INFO: 10.0.151.187:53496 - "GET /healthz HTTP/1.1" 200 OK
INFO: 10.0.151.187:39988 - "GET /healthz HTTP/1.1" 200 OK
INFO: 10.0.151.187:39992 - "GET /healthz HTTP/1.1" 200 OK
INFO: 10.0.151.187:40000 - "GET /ready HTTP/1.1" 200 OK
INFO: 10.0.152.157:41520 - "GET /v1/models HTTP/1.1" 200 OK
2026-01-14 23:03:26,396 - artifact_service.py:66 - __init__ - INFO - {"event": "Initialized ArtifactDelegationService: local_dir=/tmp/aibrix/adapters, credentials_mount=/var/run/secrets/aibrix", "logger": "aibrix.runtime.artifact_service", "level": "info", "timestamp": "2026-01-14 23:03:26 UTC"}
2026-01-14 23:03:26,396 - app.py:149 - load_lora_adapter - INFO - {"event": "Loading adapter with artifact delegation: qwen2-5-7b-lora from tos://aibrix-artifact-testing/models/Qwen2.5-7B-lora/Qwen2.5-7B-LoRA/ and request=lora_name='qwen2-5-7b-lora' artifact_url='tos://aibrix-artifact-testing/models/Qwen2.5-7B-lora/Qwen2.5-7B-LoRA/' credentials_secret=None credentials=*** additional_config=None local_dir='/tmp/aibrix/adapters'", "logger": "aibrix.app", "level": "info", "timestamp": "2026-01-14 23:03:26 UTC"}
2026-01-14 23:03:26,397 - artifact_service.py:202 - load_adapter_with_delegation - INFO - {"event": "Starting artifact delegation for adapter qwen2-5-7b-lora from tos://aibrix-artifact-testing/models/Qwen2.5-7B-lora/Qwen2.5-7B-LoRA/", "logger": "aibrix.runtime.artifact_service", "level": "info", "timestamp": "2026-01-14 23:03:26 UTC"}
2026-01-14 23:03:26,397 - artifact_service.py:212 - load_adapter_with_delegation - INFO - {"event": "Using direct credentials from request with keys: ['TOS_ACCESS_KEY', 'TOS_SECRET_KEY', 'aws_access_key_id', 'aws_secret_access_key']", "logger": "aibrix.runtime.artifact_service", "level": "info", "timestamp": "2026-01-14 23:03:26 UTC"}
2026-01-14 23:03:26,397 - artifact_service.py:153 - download_artifact - INFO - {"event": "Downloading artifact for qwen2-5-7b-lora from tos://aibrix-artifact-testing/models/Qwen2.5-7B-lora/Qwen2.5-7B-LoRA/ to /tmp/aibrix/adapters/qwen2-5-7b-lora", "logger": "aibrix.runtime.artifact_service", "level": "info", "timestamp": "2026-01-14 23:03:26 UTC"}
however the default sidecar injector would be:
LivenessProbe: &corev1.Probe{
ProbeHandler: corev1.ProbeHandler{
HTTPGet: &corev1.HTTPGetAction{
Path: SidecarHealthPath,
Port: intstr.FromInt(SidecarPort),
Scheme: corev1.URISchemeHTTP,
},
},
InitialDelaySeconds: 3,
PeriodSeconds: 2,
},
ReadinessProbe: &corev1.Probe{
ProbeHandler: corev1.ProbeHandler{
HTTPGet: &corev1.HTTPGetAction{
Path: SidecarReadyPath,
Port: intstr.FromInt(SidecarPort),
Scheme: corev1.URISchemeHTTP,
},
},
InitialDelaySeconds: 5,
PeriodSeconds: 10,
},
Steps to Reproduce
create a stormservice and deploy a modeladapter and try to download it.
apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
name: qwen2-5-7b-lora
namespace: default
labels:
model.aibrix.ai/name: "qwen2-5-7b-lora"
model.aibrix.ai/port: "8000"
spec:
baseModel: qwen2.5-7b
podSelector:
matchLabels:
model.aibrix.ai/name: qwen2.5-7b
adapter.model.aibrix.ai/enabled: "true"
artifactURL: "tos://aibrix-artifact-testing/models/Qwen2.5-7B-lora/Qwen2.5-7B-LoRA/"
credentialsSecretRef:
name: tos-credential
schedulerName: default
when downloading, it's will be kill due to unhealthy status
Expected behavior
create a separate thread pool to download, so do not block the health check.
Environment
AIBrix runtime v0.5.0
in volcano engine, download from volc tos.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with artifact_service.py, especially download_artifact and load_adapter_with_delegation, then inspect the sidecar injector's liveness and readiness probe definitions shown in the issue. Reproduce the ModelAdapter download with the provided YAML; done means artifact downloads no longer block health checks or cause the runtime to be restarted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, python
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100