InternLM / InternLM/lmdeploy

[Docs] llava-llama3的图片预处理和前向推理过程

Open
#2,309 4 comments 0 reactions 1 assignee Assigned to @irexyc View on GitHub
Dominant language
Python
Stars
8.1k
Forks
748
Avg merge
6d 2h
Merged PRs (30d)
54

Description

### 📚 The doc issue

感谢你们的工作,我参考官方文档:https://github.com/InternLM/lmdeploy/blob/main/docs/zh_cn/multi_modal/vl_pipeline.md,能够跑通,代码如下
```
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "3"
from lmdeploy import pipeline, ChatTemplateConfig
from lmdeploy.vl import load_image
pipe = pipeline('**, chat_template_config=ChatTemplateConfig(model_name='llama3'))

image = load_image('**')
response = pipe(('describe this image', image))
print(response)
```
代码能照常跑通,通过debug了解些细节, 比如lmdeploy/vl/model/llava.py:build_model函数会加载vlm函数.
原本以为preprocess是图像预处理, forward是llm的前向推理过程,但是在这两处打断点, 运行的时候并没有经过, 所以想问下图像预处理模块和llm的forward是在哪加载,因为我想修改图像预处理模块, 感谢您的答复.

### Suggest a potential alternative/fix

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.