InternLM/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Stars: 8.1k
Forks: 748
Open beginner issues: 0
Indexed issues: 546
Avg merge: 6d 2h
Merged PRs (30d): 54

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 16, 2026
Latest indexed: Sep 17, 2026
Contributing guide: Contributing guide
Code of conduct: No code of conduct
Beginner labels: No beginner labels indexed

Open All

546 open issues indexed

[Bug] llama2-7b smooth量化后推理性能未提升 Open

InternLM/lmdeploy#2331 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Feature] LLaVA 1.5 WINT 8 量化 Open

InternLM/lmdeploy#2336 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] 0.5.3 回复内容重复性高 Open

InternLM/lmdeploy#2359 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Bug] use openai server, request get asyncio.exceptions.TimeoutError Open

InternLM/lmdeploy#2360 · 3 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
deploying Mixtral8x22B with quantization Open

InternLM/lmdeploy#2376 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Feature] Reuse the text prompt encoding when processing batch jobs Open

InternLM/lmdeploy#2380 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Bug] 無法在windows上部署 Phi-3.5-vision-instruct Open

InternLM/lmdeploy#2385 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Feature] 海光DCU简单测试，希望能支持 Open

InternLM/lmdeploy#2386 · 2 comments · 1 reaction · 0 assignees · Indexed Sep 17, 2026
[Bug] error occurs when reset chat session in gradio with image uploded and get response. Open

InternLM/lmdeploy#2387 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
支持对qwen2-audio-instruct的加速吗 Open

InternLM/lmdeploy#2398 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Bug] [TM][ERROR] CUDA runtime error: misaligned address Open

InternLM/lmdeploy#2399 · 7 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Bug] 2卡internvl2-26b推理，卡间通信是pcie会失败，nvlink会成功，这是为啥 Open

InternLM/lmdeploy#2400 · 12 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Bug] Aborted (core dumped) Open

InternLM/lmdeploy#2409 · 3 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] CUDA runtime error: operation not supported /lmdeploy/src/turbomind/utils/allocator.h:197 Open

InternLM/lmdeploy#2414 · 24 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Docs] AWQ / GPTQ 部分 Open

InternLM/lmdeploy#2422 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] Profiling GeMM kernel in lmdeploy Open

InternLM/lmdeploy#2424 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] triton.runtime.autotuner.OutOfResources: out of resource: shared memory, Required: 108672, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. Open

InternLM/lmdeploy#2451 · 7 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] output not consistent with different max_prefill_token_num for long context input on pytorch engine Open

InternLM/lmdeploy#2457 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] 2x4090 with Llama2 70B silently crashes (i.e. without any error message in DEBUG mode) as of v0.6.0a0 and v0.6.0 (but works fine in previous versions) Open

InternLM/lmdeploy#2468 · 10 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] Will multi-modal models support W8A8 quantization in the future? Open

InternLM/lmdeploy#2496 · 6 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] internlm2_5-7b-chat多卡部署报错 aborted Open

InternLM/lmdeploy#2508 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Bug] llama3.1 70B v1/chat/completions error on Huawei Ascend 910B Open

InternLM/lmdeploy#2515 · 4 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] InternVL2-4B turbomind支持 Open

InternLM/lmdeploy#2524 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Bug] lmdeploy + InternVL2-40B-AWQ hangs under a certain number of asynchronous requests Open

InternLM/lmdeploy#2528 · 6 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
关于运行时候时间差异过大的问题 Open

InternLM/lmdeploy#2529 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Bug] 910b multi-card reasoning is very slow. Open

InternLM/lmdeploy#2534 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] Providing tool response back to llm for output generation is broken for llama3.1 8B Open

InternLM/lmdeploy#2542 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] Support chat completion stream with tool calls Open

InternLM/lmdeploy#2552 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] Qwen2-VL占用显存过大导致OOM Open

InternLM/lmdeploy#2565 · 8 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Docs] 问lmdeploy中的w8a8-triton实现是否有实际llm（如llama2，qwen2）的推理速度加速效果的benchmark测试？ Open

InternLM/lmdeploy#2567 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
v1/chat/interactive方式调用lmdeploy，interactive_mode=true, 图片刷新，问题不变，回答结果永远一样，这是什么原因导致的呢？ Open

InternLM/lmdeploy#2589 · 15 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] InternVL2-2B的推理速度慢，发现是视觉特征提取的耗时很长 Open

InternLM/lmdeploy#2604 · 15 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] InternVL2-26B model load extremely slow Open

InternLM/lmdeploy#2608 · 1 comment · 1 reaction · 1 assignee · Indexed Sep 17, 2026
[Bug] When TP = 4 and prefix cache is enabled, no result is generated. Open

InternLM/lmdeploy#2611 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] Internvl2-8B模型量化后推理速度减慢 Open

InternLM/lmdeploy#2616 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Bug] qwen2-vl-72b无法使用 Open

InternLM/lmdeploy#2622 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] 对InternVL模型进行推理时，图像编码阶段gpu-cpu的传输时间过长 Open

InternLM/lmdeploy#2624 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] Combine Batched Inference and Chat Conversation in VLMs Deployment Open

InternLM/lmdeploy#2628 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Bug] Phi-3-vision-128k-instruct 跑模型在8卡上出现 “Expected all tensors to be on the same device, but found at least two devices” Open

mllm

InternLM/lmdeploy#2633 · 4 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] cpu 100% Open

InternLM/lmdeploy#2637 · 3 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] Use triton to deploy minicpm-v-2_6 GPU memory keeps increasing until it overflows Open

InternLM/lmdeploy#2642 · 3 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] use new 4bits quantizated models of internlm2, decoded word starts with a blank. Open

InternLM/lmdeploy#2651 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] support multi-lora in turbomind backend Open

InternLM/lmdeploy#2674 · 0 comments · 1 reaction · 1 assignee · Indexed Sep 17, 2026
[Feature] Support QwenVL on Ascend Open

InternLM/lmdeploy#2675 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] pytorch backend 's precision points loss 1.0-2.5 points between main code and v0.6.1 on some models. Open

InternLM/lmdeploy#2679 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Docs] LoRA 推理服务 Open

InternLM/lmdeploy#2686 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Bug] Accuracy of W8A8 is big different from that of the original model Open

InternLM/lmdeploy#2730 · 6 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] The `cache-max-entry-count` working off percentages makes it difficult to setup multiple servers Open

InternLM/lmdeploy#2732 · 4 comments · 0 reactions · 1 assignee · Indexed Sep 17, 2026
[Feature] Qwen2-VL支持video Open

InternLM/lmdeploy#2735 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 17, 2026
[Feature] 有昇腾平台的模型性能测试数据吗 Open

InternLM/lmdeploy#2746 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 17, 2026

A short digest of beginner-friendly GitHub issues.