SSD tree cache KV cache
Open
- Dominant language
- C
- Stars
- 22.3k
- Forks
- 2.1k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 4
Description
Since you’re saving the KV cache to an SSD, wouldn’t it be a good idea to create a database tree, perhaps using ‘Patricia trie’ or RocksDB?
After all, system prompts are constantly repeated, and saving most of the work you’ve done would save a huge amount of inference.
Contributor guide
Assessment
This issue has not been assessed yet.