google-deepmind / google-deepmind/gemma

Proposal: Symbolic Activation Framework — Eliminate KV Cache Context Rot

Open
#666 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

Hi @yaGuangLi @aijunbai,

I'm GCat, and I've published research identifying **Context Rot**—a critical flaw in KV caching that degrades LLM performance in long conversations.

Key findings:
- After 15+ turns / 32k tokens, effective info utilization drops below 40%
- Root cause: Transformer + explicit KV cache, not long-context capability
- Solution: **Symbolic Activation Framework** → 95%+ compression, zero context pollution

Paper: https://zenodo.org/records/20433184
Code: https://github.com/gymaira1990-jpg/catnest/tree/main/00-%E8%AE%BA%E6%96%87%E5%90%88%E9%9B%86

Would be great to discuss integrating this into Gemini/Gemma inference.

Thanks!
GCat

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.