mudler / mudler/vllm.cpp

Carry per-group slot mappings into multi-KV forwards

Open
#3,109 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: MODEL-MM-deepseek-v4-deepseek-v4-for-causal-lm

DeepSeek V4 Vision W4 found that StepInputs::slot_mapping[g] is computed for every KV group, but the runner forwards only the full-attention group's mapping through CommonAttentionMetadata. MultiKvCacheIndex carries per-group block tables but no parallel per-group slot mappings.

That makes a production multi-cache model validate the real cache groups and then either invent identity slots or ignore non-primary cache writes. W4 needs the actual SWA, compressed-latent, attention-compressor, indexer-key, and indexer-compressor slots.

Fix in the same flow: add a borrowed per-group slot-mapping channel to the shared multi-KV index, populate it from runner-owned step inputs for the forward lifetime, and mutation-prove a non-identity mapping through the registered DeepSeek V4 path. The pull request for #2411 closes this issue too.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace StepInputs::slot_mapping through CommonAttentionMetadata and MultiKvCacheIndex, starting with the shared multi-KV forward flow. Determine how runner-owned mappings live for the forward and how the registered DeepSeek V4 path consumes them. Done means each KV group receives its actual mapping, and a non-identity mapping is mutation-proven through that path.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai-infra-agents, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.