[Inference] Optimize MLA latent KV-cache append with Triton
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
## Summary
The dynamic inference path uses a fused Triton append kernel for the standard KV-cache layout, but the `cache_mla_latent=True` path currently falls back to PyTorch indexed assignment when writing compressed MLA latents.
This issue proposes adding a dedicated Triton append kernel for the MLA latent cache write path.
## Scope
- Preserve the existing MLA latent cache layout.
- Preserve the current block allocator and block table semantics.
- Preserve the PyTorch fallback when Triton is unavailable.
- Do not change attention dispatch, FlashInfer integration, block-size guards, or KV-cache dtype.
- Add numerical parity tests and an append-only benchmark.
This is independent of PRs #4918, #4919, and #4920, which address MLA block-size validation, attention dispatch, and functional coverage respectively.
Contributor guide
Assessment
This issue has not been assessed yet.