[Bug] `fetch_health_check_configs_by_revision_ids` drops `initial_delay` for non-CUSTOM runtimes — large model deployments evicted mid-load
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
## Summary
In Backend.AI 26.4.x, deployment routes for **non-CUSTOM runtimes** (vllm, sglang, etc.) ignore the `initial_delay` value declared in `model_definition.health_check`. The runtime profile path is used directly with `ModelHealthCheck(path=...)`, which leaves `initial_delay` at its Pydantic default of **60.0 s**. Large models that take >60 s to load weights are flagged unhealthy by the manager and evicted at ~+4 min, well before they finish loading.
## Reproduction
1. Deploy a `vllm` model that takes >60 s to load weights (e.g. Qwen3.5-35B-A3B-NVFP4 on a single GPU node, or any 70B+ model).
2. Set `model_definition.health-check.initial-delay: 1800.0` in `deployment-config.yaml` (this is the new 26.4 default for the vllm definition_generator).
3. Verify the value is persisted: `SELECT model_definition->'health_check'->'initial_delay' FROM deployment_revisions WHERE id=...;` returns `1800.0`.
4. Watch `route:health:` in Valkey: `initial_delay_until - running_at == 60` (not 1800).
5. Manager marks the route for eviction at ~+240s while shards are still loading. Kernel log shows `check_model_health → unhealthy` at +110s; manager log shows `Marked 1 routes for eviction` at +240s.
## Root Cause
`src/ai/backend/manager/repositories/deployment/db_source/db_source.py:1903-1908`:
```python
for row in result:
profile = MODEL_SERVICE_RUNTIME_PROFILES[row.runtime_variant]
if profile.health_check_endpoint:
configs[row.id] = ModelHealthCheck(path=profile.health_check_endpoint) # 🔴 drops initial_delay
elif row.runtime_variant == "custom" and row.model_definition:
md = ModelDefinition.model_validate(row.model_definition)
configs[row.id] = md.health_check_config()
else:
configs[row.id] = None
```
Non-CUSTOM runtimes always enter the first branch (vllm/sglang/etc. profiles all set `health_check_endpoint`). `ModelHealthCheck(path=...)` does not pass `initial_delay`, so `src/ai/backend/common/config.py:236` `initial_delay: float = Field(default=60.0, ...)` applies. The yaml-supplied value persisted in `deployment_revisions.model_definition` is never consulted for non-CUSTOM runtimes.
The intended path was:
1. `definition_generator/vllm.py:31` constructs `ModelHealthCheckDraft(initial_delay=1800.0)`.
2. yaml is written → DB JSONB stored under `model_definition.health_check`.
3. On route RUNNING, `route/executor.py:258` calls `fetch_health_check_configs_by_revision_ids` and stores `initial_delay_until = running_at + int(initial_delay)`.
Step 3 silently drops the user-specified value.
## Sister Bug (same logic, app-proxy path)
`src/ai/backend/manager/registry.py:2391-2397` `get_health_check_info` has the same `runtime_variant == "custom"` gate. This affects the app-proxy path. Should be patched in the same fix.
## Impact
- Affects **all 26.4.x deployments** that use non-CUSTOM runtimes (vllm/sglang/etc.) with large models.
- Any model whose load time exceeds 60 s is killed at ~4 min into deployment, before becoming healthy.
- Defeats the purpose of the new 1800 s default in `model_definition.health-check.initial-delay`.
- Pydantic `validate_by_name=True` + `alias_generator=snake_to_kebab_case` works correctly (`ModelDefinition.model_validate(...)` parses the JSONB fine) — the bug is purely the missing call.
## Proposed Fix
Parse `model_definition` first when present, regardless of `runtime_variant`. Fall back to the runtime profile only when no model_definition is supplied:
```python
for row in result:
profile = MODEL_SERVICE_RUNTIME_PROFILES[row.runtime_variant]
if row.model_definition:
try:
md = ModelDefinition.model_validate(row.model_definition)
cfg = md.health_check_config()
if cfg is not None:
configs[row.id] = cfg
continue
except Exception as e:
log.warning(
"failed to parse model_definition for revision {}: {}",
row.id, e,
)
if profile.health_check_endpoint:
configs[row.id] = ModelHealthCheck(path=profile.health_check_endpoint)
else:
configs[row.id] = None
```
Apply the same fix to `registry.py:2391-2397`.
## Verification (hotfix applied to a customer node, 2026-04-17)
- After patching `db_source.py`, a new route's Valkey record:
- `running_at = 1776393733`
- `initial_delay_until = 1776395533`
- delta = **1800 s** ✓
- Qwen3.5-35B turned healthy at +280 s; previously evicted at +240 s.
- vLLM `GET /health 200 OK` confirmed.
## Related
- Common cause behind on-site Gemma-4 / Qwen3.5 deployment failures over the past two weeks.
- Manager log signature: `Marked N routes for eviction` after +240 s of healthy-loading kernels.
- Released in: 26.4.0~26.4.3.
Contributor guide
Research direction
Start with fetch_health_check_configs_by_revision_ids in src/ai/backend/manager/repositories/deployment/db_source/db_source.py and get_health_check_info in src/ai/backend/manager/registry.py, then trace route/executor.py and ModelDefinition.health_check_config(). Verify the persisted model_definition value is used for non-CUSTOM runtimes while profile fallback still works; done when initial_delay is preserved through route startup and the app-proxy path is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100