bug: [lmcache/lmstack-router] Router does not forward request after uncertain amount of time

Describe the bug

We started using the vllm-production-stack since 0.1.6 and at first it looked like an enormous help being able to scale multiple deployments and having the router take care about load balancing as a single entry point.

However, even with the latest builds starting from 0.1.7-dev* to 0.1.8-dev* at some point (no error in logs), the requests are not forwarded to the actual backend, even if the router tells you so.

[2025-09-19 07:18:11,808] INFO: Routing request 10d2a488ed55424a34ee296d103cb43f with session id None to http://10.42.7.92:8000 at 1758266291.8078997, process time = 0.0009 (request.py:303:vllm_router.services.request_service.request)
INFO:     10.42.0.54:39626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-09-19 07:18:14,399] DEBUG: No cache hit, forwarding request to backend (main_router.py:61:vllm_router.routers.main_router)

Previously we tried session based routing logic with x-request-id as sessionKey, but switched back to the default values (round-robin) in hope to resolve the issue - Unfortunately that also did not help.

To Reproduce

I don't think it's related to the model, because we face the same issue on different models and vLLM versions, but if somebody wants to try it out, here's the config.

modelSpec:

servingEngineSpec:
  modelSpec:
    # Text Generation
    - name: "qwen-qwen3-30b-a3b-fp8"

      modelURL: "Qwen/Qwen3-30B-A3B-Thinking-2507-FP8"

      replicaCount: 1

      # LMCache release
      tag: "v0.3.6"
      repository: "lmcache/vllm-openai"

      vllmConfig:
        enablePrefixCaching: true
        enableChunkedPrefill: flase
        extraArgs:
          - --max-model-len=102400
          - --max-num-batched-tokens=16384
          - --reasoning-parser=qwen3
          - --enable-auto-tool-choice
          - --tool-call-parser=hermes
          - --chat-template=/vllm/chat_template.jinja

      env:
        - name: VLLM_USE_FLASHINFER_SAMPLER
          value: "0"
        - name: VLLM_ENGINE_ITERATION_TIMEOUT_S
          value: "180"
        # - name: LMCACHE_LOG_LEVEL
        #   value: "DEBUG"
        # - name: VLLM_LOGGING_LEVEL
        #   value: "DEBUG"

      extraVolumes:
        - name: chat-template
          configMap:
            name: chat-template-qwen3

      extraVolumeMounts:
        - name: chat-template
          mountPath: /vllm/chat_template.jinja
          subPath: chat_template.jinja

RouterSpec:

routerSpec:
  enableRouter: true

  imagePullPolicy: IfNotPresent

  repository: "lmcache/lmstack-router"
  tag: "0.1.8.dev6-g015c06d4a.d20250912"

  # routingLogic: "session"
  # sessionKey: "x-request-id"

  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 3
    targetCPUUtilizationPercentage: 80

  resources:
    requests:
      cpu: 500m
      memory: 500Mi
    limits:
      cpu: 4
      memory: 4Gi

  ingress:
    enabled: true
    className: "nginx"

    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      nginx.ingress.kubernetes.io/proxy-body-size: 75m
      nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
      nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
      nginx.ingress.kubernetes.io/proxy-connect-timeout: "300"
      nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "300"

Expected behavior

Preferrably both:

• Correct forwarding of requests
• Error-handling if connection not established or no response from model

Additional context

No response

Start with the router log path in request.py and main_router.py, then reproduce the stalled forwarding behavior using the supplied RouterSpec and model configuration. Trace what happens after the router reports a backend target and when no backend response arrives. Done means requests are forwarded reliably and connection or backend-response failures produce an error.