AssertionError : assert issubclass(connector_cls, KVConnectorBase)

Describe the bug

servingEngineSpec:
  modelSpec:
  - name: "opt125m"
 
    repository: "lmcache/vllm-openai"
    tag: "latest-nightly"
 
    modelURL: "facebook/opt-125m"
    replicaCount: 1
 
    requestCPU: 6
    requestMemory: "16Gi"
    requestGPU: 1
 
 
    vllmConfig:
      maxModelLen: 2048
      enableChunkedPrefill: true
      enablePrefixCaching: true
 
 
    lmcacheConfig:
      enabled: true
      cpuOffloadingBufferSize: "10"
 
 
    extraVolumes:
      - name: model-storage
        emptyDir: {}
 
    extraVolumeMounts:
      - name: model-storage
        mountPath: /data
 
    hf_token: "my-hf-token"
 
    nodeSelectorTerms:
      - matchExpressions:
          - key: env
            operator: In
            values:
              - gpu

i tried using the base example, if i disble the lmcacheConfig, the model is loaded and served but with lmcacheConfig enabled i am getting this error:

kubectl logs -f release-name-opt125m-deployment-vllm-75bd78955d-cg285 -n prod
INFO 07-11 12:28:30 [__init__.py:253] Automatically detected platform cuda.
INFO 07-11 12:28:36 [api_server.py:1641] vLLM API server version 0.9.2rc2.dev167+g762be26a8
INFO 07-11 12:28:36 [cli_args.py:325] non-default args: {'host': '0.0.0.0', 'max_model_len': 2048, 'enable_prefix_caching': True, 'enable_chunked_prefill': True, 'kv_transfer_config': KVTransferConfig(kv_connector='LMCacheConnectorV1', engine_id='716946f5-c4fd-42d7-a521-2d47b602a447', kv_buffer_device='cuda', kv_buffer_size=1000000000.0, kv_role='kv_both', kv_rank=None, kv_parallel_size=1, kv_ip='127.0.0.1', kv_port=14579, kv_connector_extra_config={}, kv_connector_module_path=None)}
INFO 07-11 12:28:46 [config.py:852] This model supports multiple tasks: {'reward', 'classify', 'generate', 'embed'}. Defaulting to 'generate'.
INFO 07-11 12:28:46 [config.py:1500] Using max model len 2048
WARNING 07-11 12:28:46 [arg_utils.py:1781] Compute Capability < 8.0 is not supported by the V1 Engine. Falling back to V0.
INFO 07-11 12:28:47 [config.py:2314] Chunked prefill is enabled with max_num_batched_tokens=2048.
WARNING 07-11 12:28:47 [api_server.py:241] Found PROMETHEUS_MULTIPROC_DIR was set by user. This directory must be wiped between vLLM runs or you will find inaccurate metrics. Unset the variable and vLLM will properly handle cleanup.
INFO 07-11 12:28:47 [api_server.py:272] Started engine process with PID 67
INFO 07-11 12:28:51 [__init__.py:253] Automatically detected platform cuda.
INFO 07-11 12:28:54 [llm_engine.py:230] Initializing a V0 LLM engine (v0.9.2rc2.dev167+g762be26a8) with config: model='facebook/opt-125m', speculative_config=None, tokenizer='facebook/opt-125m', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=facebook/opt-125m, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":256,"local_cache_dir":null}, use_cached_outputs=True,
INFO 07-11 12:28:57 [cuda.py:314] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 07-11 12:28:57 [cuda.py:363] Using XFormers backend.
INFO 07-11 12:28:57 [parallel_state.py:1078] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
ERROR 07-11 12:28:58 [engine.py:458]
Traceback (most recent call last):
  File "/opt/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 446, in run_mp_engine
    engine = MQLLMEngine.from_vllm_config(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 133, in from_vllm_config
    return cls(
           ^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 87, in __init__
    self.engine = LLMEngine(*args, **kwargs)
Process SpawnProcess-1:
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 265, in __init__
    self.model_executor = executor_class(vllm_config=vllm_config)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 53, in __init__
    self._init_executor()
  File "/opt/venv/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
    self.collective_rpc("init_device")
  File "/opt/venv/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2943, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/worker/worker_base.py", line 606, in init_device
    self.worker.init_device()  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/worker/worker.py", line 193, in init_device
    init_worker_distributed_environment(self.vllm_config, self.rank,
  File "/opt/venv/lib/python3.12/site-packages/vllm/worker/worker.py", line 538, in init_worker_distributed_environment
    ensure_kv_transfer_initialized(vllm_config)
  File "/opt/venv/lib/python3.12/site-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 67, in ensure_kv_transfer_initialized
    _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v0(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 49, in create_connector_v0
    assert issubclass(connector_cls, KVConnectorBase)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 460, in run_mp_engine
    raise e from None
  File "/opt/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 446, in run_mp_engine
    engine = MQLLMEngine.from_vllm_config(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 133, in from_vllm_config
    return cls(
           ^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/engine/multiprocessing/engine.py", line 87, in __init__
    self.engine = LLMEngine(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/engine/llm_engine.py", line 265, in __init__
    self.model_executor = executor_class(vllm_config=vllm_config)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 53, in __init__
    self._init_executor()
  File "/opt/venv/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
    self.collective_rpc("init_device")
  File "/opt/venv/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2943, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/worker/worker_base.py", line 606, in init_device
    self.worker.init_device()  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/worker/worker.py", line 193, in init_device
    init_worker_distributed_environment(self.vllm_config, self.rank,
  File "/opt/venv/lib/python3.12/site-packages/vllm/worker/worker.py", line 538, in init_worker_distributed_environment
    ensure_kv_transfer_initialized(vllm_config)
  File "/opt/venv/lib/python3.12/site-packages/vllm/distributed/kv_transfer/kv_transfer_state.py", line 67, in ensure_kv_transfer_initialized
    _KV_CONNECTOR_AGENT = KVConnectorFactory.create_connector_v0(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/vllm/distributed/kv_transfer/kv_connector/factory.py", line 49, in create_connector_v0
    assert issubclass(connector_cls, KVConnectorBase)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
[rank0]:[W711 12:28:59.325833960 ProcessGroupNCCL.cpp:1476] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

also, i get this line during model loading WARNING 07-11 12:28:46 [arg_utils.py:1781] Compute Capability < 8.0 is not supported by the V1 Engine. Falling back to V0.

To Reproduce

https://github.com/vllm-project/production-stack/issues/574

after trying new docker image, i get the above error.

Expected behavior

No response

Additional context

No response

Start with the vLLM entry points shown in the traceback: distributed/kv_transfer/kv_transfer_state.py, kv_connector/factory.py, and arg_utils.py, then compare the new Docker image and the reported V0 fallback on the stated GPU. Use issue #574 as the reproduction context. Done means identifying the LMCache connector compatibility problem and making or documenting a configuration that lets the model start with lmcacheConfig enabled.