lablup / lablup/backend.ai

[Bug] `fetch_health_check_configs_by_revision_ids` drops `initial_delay` for non-CUSTOM runtimes — large model deployments evicted mid-load

Open
#11,171 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Summary

In Backend.AI 26.4.x, deployment routes for **non-CUSTOM runtimes** (vllm, sglang, etc.) ignore the `initial_delay` value declared in `model_definition.health_check`. The runtime profile path is used directly with `ModelHealthCheck(path=...)`, which leaves `initial_delay` at its Pydantic default of **60.0 s**. Large models that take >60 s to load weights are flagged unhealthy by the manager and evicted at ~+4 min, well before they finish loading.

## Reproduction

1. Deploy a `vllm` model that takes >60 s to load weights (e.g. Qwen3.5-35B-A3B-NVFP4 on a single GPU node, or any 70B+ model).
2. Set `model_definition.health-check.initial-delay: 1800.0` in `deployment-config.yaml` (this is the new 26.4 default for the vllm definition_generator).
3. Verify the value is persisted: `SELECT model_definition->'health_check'->'initial_delay' FROM deployment_revisions WHERE id=...;` returns `1800.0`.
4. Watch `route:health:` in Valkey: `initial_delay_until - running_at == 60` (not 1800).
5. Manager marks the route for eviction at ~+240s while shards are still loading. Kernel log shows `check_model_health → unhealthy` at +110s; manager log shows `Marked 1 routes for eviction` at +240s.

## Root Cause

`src/ai/backend/manager/repositories/deployment/db_source/db_source.py:1903-1908`:

```python
for row in result:
profile = MODEL_SERVICE_RUNTIME_PROFILES[row.runtime_variant]
if profile.health_check_endpoint:
configs[row.id] = ModelHealthCheck(path=profile.health_check_endpoint) # 🔴 drops initial_delay
elif row.runtime_variant == "custom" and row.model_definition:
md = ModelDefinition.model_validate(row.model_definition)
configs[row.id] = md.health_check_config()
else:
configs[row.id] = None
```

Non-CUSTOM runtimes always enter the first branch (vllm/sglang/etc. profiles all set `health_check_endpoint`). `ModelHealthCheck(path=...)` does not pass `initial_delay`, so `src/ai/backend/common/config.py:236` `initial_delay: float = Field(default=60.0, ...)` applies. The yaml-supplied value persisted in `deployment_revisions.model_definition` is never consulted for non-CUSTOM runtimes.

The intended path was:
1. `definition_generator/vllm.py:31` constructs `ModelHealthCheckDraft(initial_delay=1800.0)`.
2. yaml is written → DB JSONB stored under `model_definition.health_check`.
3. On route RUNNING, `route/executor.py:258` calls `fetch_health_check_configs_by_revision_ids` and stores `initial_delay_until = running_at + int(initial_delay)`.

Step 3 silently drops the user-specified value.

## Sister Bug (same logic, app-proxy path)

`src/ai/backend/manager/registry.py:2391-2397` `get_health_check_info` has the same `runtime_variant == "custom"` gate. This affects the app-proxy path. Should be patched in the same fix.

## Impact

- Affects **all 26.4.x deployments** that use non-CUSTOM runtimes (vllm/sglang/etc.) with large models.
- Any model whose load time exceeds 60 s is killed at ~4 min into deployment, before becoming healthy.
- Defeats the purpose of the new 1800 s default in `model_definition.health-check.initial-delay`.
- Pydantic `validate_by_name=True` + `alias_generator=snake_to_kebab_case` works correctly (`ModelDefinition.model_validate(...)` parses the JSONB fine) — the bug is purely the missing call.

## Proposed Fix

Parse `model_definition` first when present, regardless of `runtime_variant`. Fall back to the runtime profile only when no model_definition is supplied:

```python
for row in result:
profile = MODEL_SERVICE_RUNTIME_PROFILES[row.runtime_variant]
if row.model_definition:
try:
md = ModelDefinition.model_validate(row.model_definition)
cfg = md.health_check_config()
if cfg is not None:
configs[row.id] = cfg
continue
except Exception as e:
log.warning(
"failed to parse model_definition for revision {}: {}",
row.id, e,
)
if profile.health_check_endpoint:
configs[row.id] = ModelHealthCheck(path=profile.health_check_endpoint)
else:
configs[row.id] = None
```

Apply the same fix to `registry.py:2391-2397`.

## Verification (hotfix applied to a customer node, 2026-04-17)

- After patching `db_source.py`, a new route's Valkey record:
- `running_at = 1776393733`
- `initial_delay_until = 1776395533`
- delta = **1800 s** ✓
- Qwen3.5-35B turned healthy at +280 s; previously evicted at +240 s.
- vLLM `GET /health 200 OK` confirmed.

## Related

- Common cause behind on-site Gemma-4 / Qwen3.5 deployment failures over the past two weeks.
- Manager log signature: `Marked N routes for eviction` after +240 s of healthy-loading kernels.
- Released in: 26.4.0~26.4.3.

Contributor guide

Open the contributing guide

Research direction

Start with fetch_health_check_configs_by_revision_ids in src/ai/backend/manager/repositories/deployment/db_source/db_source.py and get_health_check_info in src/ai/backend/manager/registry.py, then trace route/executor.py and ModelDefinition.health_check_config(). Verify the persisted model_definition value is used for non-CUSTOM runtimes while profile fallback still works; done when initial_delay is preserved through route startup and the app-proxy path is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.