InternLM / InternLM/lmdeploy

[Bug] Issues Running Vision Language Models in Docker

Open
#1,514 8 comments 0 reactions 1 assignee Claimed by @lvhan028 View on GitHub
backlog
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### Checklist

- [X] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.

### Describe the bug

Hi folks, thanks for this great project. I am raising this issue in case it helps anyone else with Docker.

When running different models with Docker as per the instructions at [Option 2: Deploying with Docker](https://lmdeploy.readthedocs.io/en/latest/serving/api_server_vl.html#option-2-deploying-with-docker), I have run into various issues with Python dependencies. This has been resolved by creating a new `Dockerfile` with the necessary Python deps installed. For example, when running Llava 1.6 34B one runs into dependency issues such as no such module `timm`, and "install llava at git@...":

The following Dockerfile works for me for Llava:

```Docker
FROM openmmlab/lmdeploy:latest

RUN apt-get update && apt-get install -y python3 python3-pip git

WORKDIR /app

RUN pip3 install --upgrade pip
RUN pip3 install timm
RUN pip3 install git+https://github.com/haotian-liu/LLaVA.git --no-deps

COPY . .

CMD ["lmdeploy", "serve", "api_server", "liuhaotian/llava-v1.6-34b"]
```
Likewise for Yi-VL.

For Deepseek-VL, this worked:

```Docker
FROM openmmlab/lmdeploy:latest

RUN apt-get update && apt-get install -y python3 python3-pip git

WORKDIR /app

RUN pip3 install --upgrade pip
RUN pip3 install git+https://github.com/deepseek-ai/DeepSeek-VL.git --no-deps
RUN pip3 install attrdict
RUN pip3 install timm

COPY . .

CMD ["lmdeploy", "serve", "api_server", "deepseek-ai/deepseek-vl-7b-chat"]
```

### Reproduction

```bash
docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HUGGING_FACE_HUB_TOKEN=" \
-p 23333:23333 \
--ipc=host \
openmmlab/lmdeploy:latest \
lmdeploy serve api_server liuhaotian/llava-v1.6-34b
```

### Environment

```Shell
Ubuntu 22.04
```

### Error traceback

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.