Router returns fatal 404 (not retryable 503) for a scaled-to-zero model when the router pod started while the pool was at 0 replicas

Start with has_ever_seen_model() and the 404/503 branch in src/vllm_router/services/request_service/request.py, then inspect K8sPodIPServiceDiscovery and its known_models population. Compare the static model declaration and Deployment/ReplicaSet watch options described in the issue. Done means a router started while a model is scaled to zero returns retryable 503 responses consistently rather than process-dependent fatal 404s.

vllm_router decides 404-vs-503 for a model with no live endpoints using has_ever_seen_model() (src/vllm_router/services/request_service/request.py):

if not endpoints:
    if not model_ever_existed:
        return JSONResponse(status_code=404, ...)   # fatal
    else:
        return JSONResponse(status_code=503, ...)   # retryable

For K8sPodIPServiceDiscovery, has_ever_seen_model() consults self.known_models — a set populated only by pods this router process has observed. A router pod that starts (or restarts) while a model's deployment sits at 0 replicas has never observed an engine for it, so it answers a fatal 404 Model not found for a model that is merely scaled to zero. A sibling router pod started earlier answers a retryable 503 for the same request. With kube-proxy round-robin, ~50% of requests get the fatal answer.

Observed impact (autoscale-to-zero with KEDA): the 404 is self-reinforcing. A fatal 404 fails clients instantly, so they stop sustaining the request-rate metric the autoscaler needs to wake the pool — whereas 503-plus-retry keeps traffic up through the cold start. Combined with a cooldown shorter than the cold start, the model never recovers without operator action.

Numbers from one staging incident: a router pod that started while the pool was at 0 answered 92× 404 / 0× 503 over 20h; its sibling (started when engines existed) answered 466× 503 / 0× 404.

Suggested fix

Drive the 404/503 split from deployment configuration rather than per-process observation history. Options:

1. Let operators statically declare the deployment's model names (flag or env, e.g. VLLM_ROUTER_KNOWN_MODELS=modelA,modelB); has_ever_seen_model returns true for declared models. (We run this as a carried patch — happy to PR it.)
2. Have K8sPodIPServiceDiscovery watch Deployments/ReplicaSets matching the label selector (which exist at 0 replicas) rather than only Pods.

Happy to contribute either — preference?