[mooncake_backend.py:240] Failed to get 6722 keys out of 6722. error_codes=[-707]. Check key existence and memory state.

### Bug Report

(EngineCore pid=27737) ERROR 07-23 11:54:45 [mooncake_backend.py:240] Failed to get 6722 keys out of 6722. error_codes=[-707]. Check key existence and memory state.
(EngineCore pid=27737) ERROR 07-23 11:54:45 [kv_transfer.py:753] Failed to load blocks. failed_count=111, failed_blocks={0, 7526, 7527, 7528, 7529, 7530, 7531, 7532, 7533, 7534, 7535, 7536, 7537, 7538, 7539, 7540, 7541, 7542, 7543, 7544, 7545, 7546, 7547, 7548, 7549, 7550, 7551, 7552, 7553, 7554, 7555, 7556, 7557, 7558, 7559, 7560, 7561, 7562, 7563, 7564, 7565, 7566, 7567, 7568, 7569, 7570, 7571, 7572, 7573, 7574, 7575, 7576, 7577, 7578, 7579, 7580, 7581, 7582, 7583, 7584, 7585, 7586, 7587, 7588, 7589, 7590, 7591, 7592, 7593, 7594, 7595, 7596, 7597, 7598, 7599, 7600, 7601, 7602, 7603, 7604, 7605, 7606, 7607, 7608, 7609, 7610, 7611, 7612, 7613, 7614, 7615, 7616, 7617, 7618, 7619, 7620, 7621, 7622, 7623, 7624, 7625, 7626, 7627, 7628, 7629, 7630, 7631, 7632, 7633, 7634, 7635}. Check block availability and memory state.
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] EngineCore encountered a fatal error.
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] Traceback (most recent call last):
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1158, in run_engine_core
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] engine_core.run_busy_loop()
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1199, in run_busy_loop
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] self._process_engine_step()
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1238, in _process_engine_step
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] outputs, model_executed = self.step_fn()
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 557, in step_with_batch_queue
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] engine_core_outputs = self.scheduler.update_from_output(
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py", line 1316, in update_from_output
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] failed_kv_load_req_ids = self._handle_invalid_blocks(
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py", line 2299, in _handle_invalid_blocks
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] self._update_requests_with_invalid_blocks(
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/core/sched/scheduler.py", line 2208, in _update_requests_with_invalid_blocks
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] (req_block_ids,) = self.kv_cache_manager.get_block_ids(req_id)
(EngineCore pid=27737) ERROR 07-23 11:54:49 [core.py:1167] ValueError: too many values to unpack (expected 1)
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] AsyncLLM output_handler failed.
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] Traceback (most recent call last):
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 660, in output_handler
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] outputs = await engine_core.get_output_async()
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] File "/usr/local/python3.10.7/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 1030, in get_output_async
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] raise self._format_exception(outputs) from None
(APIServer pid=27578) ERROR 07-23 11:54:49 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

mooncake version:latest
vllm version: 0.22.1+empty
1.3 Kvcache pooling
P node
export MOONCAKE_CONFIG_PATH="/path/to/mooncake.json"
export PYTHONHASHSEED=0
vllm serve /path/to/DeepSeek-V4-Flash/ --max_model_len 65536 --no-enable-prefix-caching --block_size 2 --max-num-batched-tokens 4096 --served-model-name qwen --gpu-memory-utilization 0.95 --async-scheduling --max-num-seqs 64 --port 8555 -tp 1 --additional_config '{"enable_cpu_binding": "True"}' --load_format safetensors --compilation-config '{"cudagraph_capture_sizes":[8], "cudagraph_mode": "FULL_DECODE_ONLY"}' --kv-transfer-config '{"kv_connector": "MultiConnector", "kv_role": "kv_producer", "kv_connector_extra_config": { "connectors": [ { "kv_connector": "MooncakeConnector", "kv_role": "kv_producer", "kv_port": "30200", "kv_connector_module_path": "vllm_ascend.distributed.kv_transfer.kv_p2p.mooncake_connector", "kv_connector_extra_config": { "prefill": {"dp_size": 1, "tp_size": 1}, "decode": {"dp_size": 1, "tp_size": 1} } }, { "kv_connector": "AscendStoreConnector", "kv_role": "kv_producer", "kv_connector_extra_config": { "lookup_rpc_port": "0", "backend": "mooncake" } } ] } }' --enforce-eager

D node
export MOONCAKE_CONFIG_PATH="/path/to/mooncake.json"
export PYTHONHASHSEED=0

vllm serve /path/to/DeepSeek-V4-Flash/ --max_model_len 65536 --no-enable-prefix-caching --block_size 2 --max-num-batched-tokens 4096 --served-model-name qwen --gpu-memory-utilization 0.95 --async-scheduling --max-num-seqs 64 --port 8555 -tp 1 --additional_config '{"enable_cpu_binding": "True"}' --load_format safetensors --compilation-config '{"cudagraph_capture_sizes":[8], "cudagraph_mode": "FULL_DECODE_ONLY"}' --enforce-eager --kv-transfer-config '{"kv_connector": "MultiConnector", "kv_role": "kv_consumer", "kv_connector_extra_config": { "connectors": [ { "kv_connector": "MooncakeConnector", "kv_role": "kv_consumer", "kv_port": "30300", "kv_connector_module_path": "vllm_ascend.distributed.kv_transfer.kv_p2p.mooncake_connector", "kv_connector_extra_config": { "prefill": {"dp_size": 1, "tp_size": 1}, "decode": {"dp_size": 1, "tp_size": 1} } }, { "kv_connector": "AscendStoreConnector", "kv_role": "kv_consumer", "kv_connector_extra_config": { "lookup_rpc_port": "0", "backend": "mooncake" } } ] } }'

ascend_for_lingqu/examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py --host xx.xx.xx.xx --port 1025 --prefiller-hosts xx.xx.xx.xx --prefiller-port 8555 --decoder-hosts xx.xx.xx.xx --decoder-ports 8555
mooncake.json:
{
"metadata_server": "P2PHANDSHAKE",
"protocol": "ascend",
"device_name": "",
"use_ascend_direct": true,
"alloc_in_same_node": true,
"master_server_address": "xx.xx.xx.xx:50088",
"global_segment_size": 30000000000,
"local_buffer_size": 21474836480,
"default_kv_lease_ttl": 300000,
"default_kv_soft_pin_ttl": 300000,
"transfer_timeout": 10
}

mooncake_master --port 50088 --default_kv_lease_ttl 300000 --default_kv_soft_pin_ttl 300000

curl -X POST http://xx.xx.xx.xx:1025/v1/completions -H "Content-Type: application/json" -d '{
"prompt": "Something longer than 9000 tokens",
"max_tokens": 4
}'

### Before submitting...

- [ ] Ensure you searched for relevant issues and read the [documentation]

Start with mooncake_backend.py:240 and kv_transfer.py:753, then trace the scheduler paths at scheduler.py:2299 and scheduler.py:2208 shown in the traceback. Reproduce the supplied vLLM Mooncake configuration and inspect the failed block handling. Done means the reported blocks are handled without the ValueError and EngineCore no longer terminates.

Written by the indexing model from the issue text.