NVIDIA / NVIDIA/Megatron-LM

[Inference] Optimize MLA latent KV-cache append with Triton

Open
#5,820 0 comments 0 reactions 1 assignee Claimed by @sidsingh-nvidia View on GitHub
community-request enhancement
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

## Summary

The dynamic inference path uses a fused Triton append kernel for the standard KV-cache layout, but the `cache_mla_latent=True` path currently falls back to PyTorch indexed assignment when writing compressed MLA latents.

This issue proposes adding a dedicated Triton append kernel for the MLA latent cache write path.

## Scope

- Preserve the existing MLA latent cache layout.
- Preserve the current block allocator and block table semantics.
- Preserve the PyTorch fallback when Triton is unavailable.
- Do not change attention dispatch, FlashInfer integration, block-size guards, or KV-cache dtype.
- Add numerical parity tests and an append-only benchmark.

This is independent of PRs #4918, #4919, and #4920, which address MLA block-size validation, attention dispatch, and functional coverage respectively.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.