SSD tree cache KV cache
Ouverte
- Langage dominant
- C
- Étoiles
- 22.3k
- Forks
- 2.1k
- Merge moyen
- 1 j 3 h
- PR mergées (30 j)
- 4
Description
Since you’re saving the KV cache to an SSD, wouldn’t it be a good idea to create a database tree, perhaps using ‘Patricia trie’ or RocksDB?
After all, system prompts are constantly repeated, and saving most of the work you’ve done would save a huge amount of inference.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.