SSD tree cache KV cache
Abierto
- Lenguaje dominante
- C
- Estrellas
- 22.3k
- Forks
- 2.1k
- Merge medio
- 1 d 3 h
- PR fusionados (30 d)
- 4
Descripción
Since you’re saving the KV cache to an SSD, wouldn’t it be a good idea to create a database tree, perhaps using ‘Patricia trie’ or RocksDB?
After all, system prompts are constantly repeated, and saving most of the work you’ve done would save a huge amount of inference.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.