InternLM/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Stars: 8.1k
Forks: 748
Open beginner issues: 0
Indexed issues: 546
Avg merge: 6d 2h
Merged PRs (30d): 54

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 16, 2026
Latest indexed: Sep 17, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Open All

546 open issues indexed

[Bug] lora微调后一直重复输出 Open

InternLM/lmdeploy#2757 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] The script "profile_generation.py" went haywire and crashed Open

InternLM/lmdeploy#2760 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] W4A8-FP8 support in AWQ quantization Open

InternLM/lmdeploy#2766 · 3 comments · 1 reaction · 1 assignee · Indexed Sep 17, 2026
How can I specify the rope scaling type when starting the API server? Open

InternLM/lmdeploy#2768 · 1 comment · 1 reaction · 1 assignee · Indexed Sep 17, 2026
[Bug] 昇腾910B通过lmdeploy镜像，使用qwen2-vl-7b模型，推理过程报错： call aclnnBatchMatMul failed Open

InternLM/lmdeploy#2769 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] The quantization process of Qwen/Qwen2-VL-7B-Instruct is getting killed without throwing error. Open

InternLM/lmdeploy#2770 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] qwen2 vl support the turbomind engine Open

InternLM/lmdeploy#2774 · 7 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] SystemExit: 1 asyncio.exceptions.TimeoutError Open

InternLM/lmdeploy#2782 · 15 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] Response of converted Qwen2-57B-A14B-Instruct-GPTQ-Int4 returns garbled characters Open

InternLM/lmdeploy#2785 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] Llama-3.2-1B-Instruct and InternVL2-1B does not supported kvin4, is that expected? Open

InternLM/lmdeploy#2786 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] Does PytorchEngine Visual Model Support Prefix Caching? Open

planned feature

InternLM/lmdeploy#2789 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] 可以考虑支持THUDM/cogvlm2-llama3-chinese-chat-19B-int4模型吗 Open

InternLM/lmdeploy#2799 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] Why the example/cpp/llama/llama_triton_example.cc does not call the LlamaTritonModel::createEngine()? Open

InternLM/lmdeploy#2803 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] 怎么通过python代码直接获取命中的prefix前缀长度，而不进行推理等计算 Open

InternLM/lmdeploy#2808 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Feature] 想尝试将cogvideox的模型转成lmdeploy，请问这操作可能实现吗？ Open

InternLM/lmdeploy#2821 · 4 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] 4-bit KV cache results in very low performance while 8-bit KV cache is almost lossless Open

InternLM/lmdeploy#2822 · 7 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] 运行batchsize异常 Open

InternLM/lmdeploy#2827 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] Llama 3.1 API Performance Degradation (lmdeploy, api_server) After 300-400 Calls with High GPU Usage and Delayed Responses (3000s) Open

InternLM/lmdeploy#2837 · 9 comments · 1 reaction · 1 assignee · Indexed Sep 17, 2026
[Bug] chat with converted DeepSeek-V2-Lite-Chat model, raise RuntimeError Open

InternLM/lmdeploy#2840 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
How to deploy model with its lora adapter Open

InternLM/lmdeploy#2852 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
LLaVA INT4 quantization Open

InternLM/lmdeploy#2857 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
Qwen2-VL-72B-Instruct-AWQ 推理结果异常 Open

InternLM/lmdeploy#2863 · 3 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
关于lmdeploy的模型细节 Open

InternLM/lmdeploy#2871 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] llama3.2 -11b-version batch infer text-only item return Garbled code Open

InternLM/lmdeploy#2878 · 10 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
是否可以提供一个用来监控异步引擎死活的方法 Open

InternLM/lmdeploy#2895 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] Any plans on supporting `deepseek-ai/deepseek-vl2` ? Open

InternLM/lmdeploy#2897 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] InternLM2.5-20b-chat 长文本推理启动报错 `Illegal instruction` Open

InternLM/lmdeploy#2900 · 4 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] 希望加入生成参数allowed_token_ids Open

InternLM/lmdeploy#2905 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
[Bug] glm-4v图像理解在传json情况下生成乱码 Open

InternLM/lmdeploy#2909 · 3 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] vl_async_engine中存在一个拼写错误 Open

InternLM/lmdeploy#2913 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] how to apply a custom calibration data set during awq quant? Open

InternLM/lmdeploy#2923 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
部署后，连续发送请求，推理耗时递增 Open

InternLM/lmdeploy#2924 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Feature] Support deploy model on specific device id Open

InternLM/lmdeploy#2925 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Feature] pipeline.get_logits() support tensor on gpu as input rather than list input Open

InternLM/lmdeploy#2926 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] 请问如何使用自己的数据集进行lmdeploy的awq量化？量化的校对数据集只支持文字问答数据集吗？ Open

InternLM/lmdeploy#2931 · 7 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] deploy InterVL2.5 8B，Aborted (core dumped) Open

InternLM/lmdeploy#2932 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] 如何在推理时输出隐藏层 Open

InternLM/lmdeploy#2937 · 5 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] Control over Prefix Cache Capacity and Guaranteed Caching Open

InternLM/lmdeploy#2942 · 1 comment · 2 reactions · 0 assignees · Indexed Sep 17, 2026
910B 起服务失败 Open

InternLM/lmdeploy#2945 · 6 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] generation profile hangs on Mixtral-8x7B-Instruct-v0.1 with pytorch backend Open

InternLM/lmdeploy#2948 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] Support Medusa decode (Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads) Open

InternLM/lmdeploy#2950 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Feature] W8A8 support for turbomind engine Open

InternLM/lmdeploy#2962 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
昇腾lmdeploy使用 lmdeploy APIClient 接口时，推理结果被截断 Open

InternLM/lmdeploy#2969 · 5 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] 随着时间运行，lmdeploy进程占用cpu提高且不释放 Open

InternLM/lmdeploy#2990 · 5 comments · 1 reaction · 1 assignee · Indexed Sep 17, 2026
[Bug] 使用lmdeploy0.6.4部署llava-hf/llava-v1.6-vicuna-13b-hf之后，openai调用输出为空字符串 Open

InternLM/lmdeploy#3003 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] CUDA error: an illegal memory access was encountered. Vicuna results wrong Open

InternLM/lmdeploy#3004 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] OOM in jetson but not in x86 Open

InternLM/lmdeploy#3006 · 6 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] It seems the second round of answers lost the context of the first round in baichun2-13b-chat models with pytorch backend or CodeLlama-7b-Instruct-hf with turbomind backend Open

InternLM/lmdeploy#3015 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] The output cannot stop and there are a lot of repetitions on OpenGVLab/Mini-InternVL-Chat-2B-V1-5-inner-4bits Open

InternLM/lmdeploy#3022 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Bug] Tokenizer Parallelism Leads to (500 : Internal Server Error) Open

InternLM/lmdeploy#3025 · 2 comments · 1 reaction · 1 assignee · Indexed Sep 17, 2026

A short digest of beginner-friendly GitHub issues.