google-deepmind / google-deepmind/gemma
Proposal: Symbolic Activation Framework — Eliminate KV Cache Context Rot
- Dominant language
- Python
- Stars
- 5.7k
- Forks
- 1k
- Avg merge
- 10h 33m
- Merged PRs (30d)
- 2
Description
Hi @yaGuangLi @aijunbai,
I'm GCat, and I've published research identifying **Context Rot**—a critical flaw in KV caching that degrades LLM performance in long conversations.
Key findings:
- After 15+ turns / 32k tokens, effective info utilization drops below 40%
- Root cause: Transformer + explicit KV cache, not long-context capability
- Solution: **Symbolic Activation Framework** → 95%+ compression, zero context pollution
Paper: https://zenodo.org/records/20433184
Code: https://github.com/gymaira1990-jpg/catnest/tree/main/00-%E8%AE%BA%E6%96%87%E5%90%88%E9%9B%86
Would be great to discuss integrating this into Gemini/Gemma inference.
Thanks!
GCat
Contributor guide
Assessment
This issue has not been assessed yet.