vllm-project / vllm-project/aibrix

Use sidecar to download artifact will block the health check and result in restart

Open
#1,890 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.1k
Forks
697
Avg merge
1d 19h
Merged PRs (30d)
104

Description

🐛 Describe the bug

related to #1826, when use the sidecar to download the artifact from the s3 like storage it will spend some time, for example:

2026-01-14 23:03:15,548 - app.py:318 - main - INFO - {"positional_args": ["Namespace(host=None, port=8080, enable_fastapi_docs=False)"], "event": "Use %s to startup runtime server", "logger": "aibrix.app", "level": "info", "timestamp": "2026-01-14 23:03:15 UTC"}
2026-01-14 23:03:15,548 - app.py:91 - mount_metrics - INFO - {"event": "AIBrix to use /tmp/aibrix/metrics/ as PROMETHEUS_MULTIPROC_DIR", "logger": "aibrix.app", "level": "info", "timestamp": "2026-01-14 23:03:15 UTC"}
2026-01-14 23:03:15,549 - app.py:103 - mount_metrics - INFO - {"event": "AIBrix to scrape metrics from http://localhost:8000/metrics, use vllm standard rules", "logger": "aibrix.app", "level": "info", "timestamp": "2026-01-14 23:03:15 UTC"}
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     10.0.151.187:53470 - "GET /healthz HTTP/1.1" 200 OK
INFO:     10.0.151.187:53486 - "GET /healthz HTTP/1.1" 200 OK
INFO:     10.0.151.187:53496 - "GET /healthz HTTP/1.1" 200 OK
INFO:     10.0.151.187:39988 - "GET /healthz HTTP/1.1" 200 OK
INFO:     10.0.151.187:39992 - "GET /healthz HTTP/1.1" 200 OK
INFO:     10.0.151.187:40000 - "GET /ready HTTP/1.1" 200 OK
INFO:     10.0.152.157:41520 - "GET /v1/models HTTP/1.1" 200 OK
2026-01-14 23:03:26,396 - artifact_service.py:66 - __init__ - INFO - {"event": "Initialized ArtifactDelegationService: local_dir=/tmp/aibrix/adapters, credentials_mount=/var/run/secrets/aibrix", "logger": "aibrix.runtime.artifact_service", "level": "info", "timestamp": "2026-01-14 23:03:26 UTC"}
2026-01-14 23:03:26,396 - app.py:149 - load_lora_adapter - INFO - {"event": "Loading adapter with artifact delegation: qwen2-5-7b-lora from tos://aibrix-artifact-testing/models/Qwen2.5-7B-lora/Qwen2.5-7B-LoRA/ and request=lora_name='qwen2-5-7b-lora' artifact_url='tos://aibrix-artifact-testing/models/Qwen2.5-7B-lora/Qwen2.5-7B-LoRA/' credentials_secret=None credentials=*** additional_config=None local_dir='/tmp/aibrix/adapters'", "logger": "aibrix.app", "level": "info", "timestamp": "2026-01-14 23:03:26 UTC"}
2026-01-14 23:03:26,397 - artifact_service.py:202 - load_adapter_with_delegation - INFO - {"event": "Starting artifact delegation for adapter qwen2-5-7b-lora from tos://aibrix-artifact-testing/models/Qwen2.5-7B-lora/Qwen2.5-7B-LoRA/", "logger": "aibrix.runtime.artifact_service", "level": "info", "timestamp": "2026-01-14 23:03:26 UTC"}
2026-01-14 23:03:26,397 - artifact_service.py:212 - load_adapter_with_delegation - INFO - {"event": "Using direct credentials from request with keys: ['TOS_ACCESS_KEY', 'TOS_SECRET_KEY', 'aws_access_key_id', 'aws_secret_access_key']", "logger": "aibrix.runtime.artifact_service", "level": "info", "timestamp": "2026-01-14 23:03:26 UTC"}
2026-01-14 23:03:26,397 - artifact_service.py:153 - download_artifact - INFO - {"event": "Downloading artifact for qwen2-5-7b-lora from tos://aibrix-artifact-testing/models/Qwen2.5-7B-lora/Qwen2.5-7B-LoRA/ to /tmp/aibrix/adapters/qwen2-5-7b-lora", "logger": "aibrix.runtime.artifact_service", "level": "info", "timestamp": "2026-01-14 23:03:26 UTC"}

however the default sidecar injector would be:

		LivenessProbe: &corev1.Probe{
			ProbeHandler: corev1.ProbeHandler{
				HTTPGet: &corev1.HTTPGetAction{
					Path:   SidecarHealthPath,
					Port:   intstr.FromInt(SidecarPort),
					Scheme: corev1.URISchemeHTTP,
				},
			},
			InitialDelaySeconds: 3,
			PeriodSeconds:       2,
		},
		ReadinessProbe: &corev1.Probe{
			ProbeHandler: corev1.ProbeHandler{
				HTTPGet: &corev1.HTTPGetAction{
					Path:   SidecarReadyPath,
					Port:   intstr.FromInt(SidecarPort),
					Scheme: corev1.URISchemeHTTP,
				},
			},
			InitialDelaySeconds: 5,
			PeriodSeconds:       10,
		},
Steps to Reproduce

create a stormservice and deploy a modeladapter and try to download it.

apiVersion: model.aibrix.ai/v1alpha1
kind: ModelAdapter
metadata:
  name: qwen2-5-7b-lora
  namespace: default
  labels:
    model.aibrix.ai/name: "qwen2-5-7b-lora"
    model.aibrix.ai/port: "8000"
spec:
  baseModel: qwen2.5-7b
  podSelector:
    matchLabels:
      model.aibrix.ai/name: qwen2.5-7b
      adapter.model.aibrix.ai/enabled: "true"
  artifactURL: "tos://aibrix-artifact-testing/models/Qwen2.5-7B-lora/Qwen2.5-7B-LoRA/"
  credentialsSecretRef:
    name: tos-credential
  schedulerName: default

when downloading, it's will be kill due to unhealthy status

Image
Expected behavior

create a separate thread pool to download, so do not block the health check.

Environment

AIBrix runtime v0.5.0
in volcano engine, download from volc tos.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with artifact_service.py, especially download_artifact and load_adapter_with_delegation, then inspect the sidecar injector's liveness and readiness probe definitions shown in the issue. Reproduce the ModelAdapter download with the provided YAML; done means artifact downloads no longer block health checks or cause the runtime to be restarted.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, python
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.