vllm-project / vllm-project/production-stack

Lmcache Dashboard problem

Open
#818 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2.6k
Forks
503
Avg merge
4d 17h
Merged PRs (30d)
8

Description

Describe the bug

while number lmcache:num_lookup_hits_total increase in metrics
lmcache:num_hit_tokens_total is 0
which is what got in lmcache dashboard

To Reproduce

I run lmcache with this config

    - name: "qwen3-coder-30b"
      repository: "192.168.1.173:5555/lmcache/vllm-openai"
      tag: "v0.3.13"
      modelURL: "/data/hub/models--Qwen--Qwen3-Coder-30B-A3B-Instruct/snapshots/b2cff646eb4bb1d68355c01b18ae02e7cf42d120"
      imagePullPolicy: "IfNotPresent" # or "Always"
      replicaCount: 1
      requestCPU: 8
      requestMemory: "100Gi"
      requestGPU: 1
      env:
        - name: PYTORCH_ALLOC_CONF
          value: "expandable_segments:True"
        - name: LMCACHE_LOCAL_CPU
          value: "True"
        - name: LMCACHE_MAX_LOCAL_CPU_SIZE
          value: "50"
        - name: LMCACHE_CHUNK_SIZE
          value: "256"
        - name: LMCACHE_REMOTE_SERDE
          value: "cachegen"

      shmSize: "40Gi"
      vllmConfig:
        enablePrefixCaching: true
        gpuMemoryUtilization: 0.95
        maxModelLen: 32000
        extraArgs:
          - "--dtype"
          - "auto"
          - "--disable-custom-all-reduce"
          - "--trust-remote-code"
          - "--served-model-name"
          - "Qwen/Qwen3-Coder-30B-A3B-Instruct"
          - "--enable-auto-tool-choice"
          - "--tool-call-parser"
          - "hermes"
          - "--kv-transfer-config"
          - '{"kv_connector":"LMCacheConnectorV1", "kv_role":"kv_both"}'
      lmcacheConfig:
        enabled: true

      pvcStorage: "120Gi"
      pvcAccessMode:
        - ReadWriteMany

and it is log of that when I query (It is wierd why number is negetive)

(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,467] LMCache DEBUG: Looking up cache for the first time for request chatcmpl-222b7c1758ac04c5b0a5f5e3121de292-a858b6d7! (vllm_v1_adapter.py:1240:lmcache.integration.vllm.vllm_v1_adapter)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,467] LMCache DEBUG: Registered pinned object 126212516121488 for timeout monitoring at time 1770030465.47 (pin_monitor.py:92:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,468] LMCache DEBUG: Registered pinned object 126211752870784 for timeout monitoring at time 1770030465.47 (pin_monitor.py:92:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,468] LMCache DEBUG: Registered pinned object 126211752876640 for timeout monitoring at time 1770030465.47 (pin_monitor.py:92:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,468] LMCache DEBUG: Registered pinned object 126211752877456 for timeout monitoring at time 1770030465.47 (pin_monitor.py:92:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,468] LMCache DEBUG: Registered pinned object 126211752878752 for timeout monitoring at time 1770030465.47 (pin_monitor.py:92:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,468] LMCache DEBUG: Registered pinned object 126211752879520 for timeout monitoring at time 1770030465.47 (pin_monitor.py:92:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,468] LMCache DEBUG: Registered pinned object 126211746063104 for timeout monitoring at time 1770030465.47 (pin_monitor.py:92:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,468] LMCache DEBUG: Registered pinned object 126212514329040 for timeout monitoring at time 1770030465.47 (pin_monitor.py:92:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,468] LMCache INFO: Reqid: chatcmpl-222b7c1758ac04c5b0a5f5e3121de292-a858b6d7, Total tokens 2151, LMCache hit tokens: 2048, need to load: -80 (vllm_v1_adapter.py:1283:lmcache.integration.vllm.vllm_v1_adapter)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,474] LMCache DEBUG: Unregistered pinned object 126212516121488 from timeout monitoring (pin_monitor.py:104:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,474] LMCache DEBUG: Unregistered pinned object 126211752870784 from timeout monitoring (pin_monitor.py:104:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,474] LMCache DEBUG: Unregistered pinned object 126211752876640 from timeout monitoring (pin_monitor.py:104:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,474] LMCache DEBUG: Unregistered pinned object 126211752877456 from timeout monitoring (pin_monitor.py:104:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,474] LMCache DEBUG: Unregistered pinned object 126211752878752 from timeout monitoring (pin_monitor.py:104:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,474] LMCache DEBUG: Unregistered pinned object 126211752879520 from timeout monitoring (pin_monitor.py:104:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,474] LMCache DEBUG: Unregistered pinned object 126211746063104 from timeout monitoring (pin_monitor.py:104:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,475] LMCache DEBUG: Unregistered pinned object 126212514329040 from timeout monitoring (pin_monitor.py:104:lmcache.v1.pin_monitor)
(EngineCore_DP0 pid=130) [2026-02-02 11:07:45,909] LMCache DEBUG: In connector.start_load_kv, but the attn_metadata is None (vllm_v1_adapter.py:766:lmcache.integration.vllm.vllm_v1_adapter)
(APIServer pid=1) INFO:     78.39.182.215:43374 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     78.39.182.215:43376 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     78.39.182.215:43382 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.155.36:50498 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO 02-02 11:07:53 [loggers.py:257] Engine 000: Avg prompt throughput: 215.1 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 83.4%, External prefix cache hit rate: 0.0%
(APIServer pid=1) INFO:     192.168.154.84:35284 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     78.39.182.215:41128 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     78.39.182.215:41132 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.156.36:42712 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     78.39.182.215:41146 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.156.16:59148 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO 02-02 11:08:03 [loggers.py:257] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 83.4%, External prefix cache hit rate: 0.0%
(APIServer pid=1) INFO:     78.39.182.215:35040 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     78.39.182.215:35050 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.155.36:39406 - "GET /metrics HTTP/1.1" 200 OK
(EngineCore_DP0 pid=130) [2026-02-02 11:08:08,880] LMCache DEBUG: PinMonitor check: pinned_objects=0, timeout_objects=0, force_unpin_success=0 (pin_monitor.py:148:lmcache.v1.pin_monitor)
(APIServer pid=1) INFO:     192.168.154.84:38144 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     78.39.182.215:35062 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     192.168.156.36:42276 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     78.39.182.215:37066 - "GET /health HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     78.39.182.215:37074 - "GET /health HTTP/1.1" 200 OK
Expected behavior

increase num_hit_tokens_total or get another thing in dashboard

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how lmcache:num_lookup_hits_total and lmcache:num_hit_tokens_total are defined and exposed through the /metrics endpoint, then compare those values with the dashboard queries. Reproduce the provided vLLM and LMCache configuration and logs; done means cache-hit tokens are represented correctly in the dashboard or the metric behavior is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
devops, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.