KV cache connector hangs with Eagle speculative decoding when rewind crosses block boundary

Description

When using a KV cache connector with Eagle speculative decoding, the executor can hang during get_finished. The hang occurs when rejected draft tokens cause rewindKVCache to free a block that crosses a block boundary.

Root cause

KvCacheConnectorSchedulerOutputRequest.block_ids only grows monotonically — it is never trimmed after rewind. When rewindKVCache calls releaseLastBlock (which happens when the rewind crosses a block boundary), the connector retains stale references to the now-freed block ids.

This causes get_finished's cross-rank mpi_allgather + set.intersection to never complete, because the freed block ids are still present in the connector's per-request bookkeeping.

Repro

• Backend: PyTorch
• Spec decoding: Eagle (non-linear tree)
• KV cache connector: enabled (e.g. KVBM)
• Observe: hang after the first iteration where Eagle rejects draft tokens and the rewind crosses a block boundary

Suggested fix

Add an on_rewind hook to the connector that trims block_ids (and related per-request state) to match the post-rewind live cache indices after each rewindKVCache call.

Start by tracing rewindKVCache, releaseLastBlock, and get_finished, then inspect KvCacheConnectorSchedulerOutputRequest.block_ids and its per-request bookkeeping. Reproduce the Eagle speculative-decoding case with a KV cache connector enabled and observe the cross-rank hang. Done means rewinding across a block boundary leaves only live block references and get_finished completes.