SnapKV

Hello,

I see SnapKV is used for the Heavy Hitter Prompt Compression strategy. As far as I understand (correct me if I'm wrong), it is also used in the benchmarks reported in the blog post for the Heavy Hitter results.

If that's the case, I think the comparison with L2 norm, recent global etc. is unfair. In SnapKV you use the latest tokens to filter out the KV pairs that won't be used in the next tokens. However this process cannot be re-used if you ask a second question to your LLM.

For instance if you have a document, ask a first question related to the beginning of the document, SnapKV will retrieve the KV pairs at the beginning of the document. If you then ask a second question, you would have to re-run SnapKV to retrieve the KV pairs at the end of the document. Hence SnapKV does not really compress the KV cache as opposed for instance to L2 norm which definitely deletes the KV pairs. SnapKV is great to to accelerate generation, not to reduce the KV cache size in memory (except if the use case is a single interaction with the LLM).

I would make this point more clear in the blog post, or use another strategy for prompt compression heavy hitter (H2O, scissorshands etc.).

Maybe it would be worth distinguishing the context and the task in the evals ? e.g. your compress the context and the ask the question (work only if question comes after the context of)

No files or tests are named. Start by reviewing the benchmark setup and blog material discussing SnapKV and the Heavy Hitter results, then verify whether the evaluation distinguishes context compression from task-time retrieval. Done means either clarifying that limitation in the blog post and evals or documenting a decided alternative strategy.

Written by the indexing model from the issue text.