InternLM / InternLM/lmdeploy

[Feature] Support LMCache

Open
#3,860 6 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Motivation

LMDeploy has incredible throughput on smaller GPU's, it far outperforms VLLM with a much simpler setup (IMO).
Because of this it is a favourite for smaller GPU's, the kind of GPU's where offloading of KV Cache to CPU / NVMe would be beneficial.

Supporing LMCache would be great for this (and the name matches!)

https://docs.lmcache.ai/

VLLM and other inference servers already support this (or have implemented their own).

Can we get this added?

### Related resources

https://docs.vllm.ai/en/stable/examples/others/lmcache.html?h=lmcache

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.