open-compass / open-compass/VLMEvalKit

lmdeploy support Qwen3-VL-4B-Instruct

Open
#1,277 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.4k
Forks
768
Avg merge
1d 10h
Merged PRs (30d)
17

Description

commands:
lmdeploy serve api_server Qwen/Qwen3-VL-4B-Instruct --server-port 23333

Image

`Package Version Editable project location


absl-py 2.3.1
accelerate 1.10.1
addict 2.4.0
aiofiles 24.1.0
aiohappyeyeballs 2.6.1
aiohttp 3.12.15
aiosignal 1.4.0
airportsdata 20250909
aliyun-python-sdk-core 2.16.0
aliyun-python-sdk-kms 2.16.5
annotated-types 0.7.0
anthropic 0.69.0
antlr4-python3-runtime 4.9.3
anyio 4.11.0
apex 0.1
astor 0.8.1
asttokens 3.0.0
attrdict 2.0.1
attrs 25.3.0
audioread 3.0.1
auto_gptq 0.7.1
autoawq 0.2.9
av 15.1.0
binpacking 1.5.2
bitsandbytes 0.48.0
blake3 1.0.7
blobfile 3.0.0
boto3 1.40.42
botocore 1.40.42
Brotli 1.1.0
build 1.3.0
cachetools 6.2.0
cbor2 5.7.0
certifi 2025.8.3
cffi 2.0.0
cfgv 3.4.0
charset-normalizer 3.4.3
click 8.3.0
cloudpickle 3.1.1
colorama 0.4.6
colorlog 6.9.0
comm 0.2.3
compressed-tensors 0.10.2
contourpy 1.3.3
cpm-kernels 1.0.11
crcmod 1.7
cryptography 46.0.2
cuda-bindings 13.0.1
cuda-pathfinder 1.3.0
cuda-python 13.0.1
cupy-cuda12x 13.6.0
cycler 0.12.1
dacite 1.9.2
datasets 3.6.0
debugpy 1.8.17
decorator 5.2.1
decord 0.6.0
deepspeed 0.17.6
depyf 0.19.0
diffusers 0.35.1
dill 0.3.8
diskcache 5.6.3
distlib 0.4.0
distro 1.9.0
dnspython 2.8.0
docstring_parser 0.17.0
dotenv 0.9.9
e 1.4.5
editdistance 0.8.1
einops 0.8.1
email-validator 2.3.0
et_xmlfile 2.0.0
evalscope 1.0.2
evaluate 0.4.6
executing 2.2.0
fastapi 0.118.0
fastapi-cli 0.0.13
fastapi-cloud-cli 0.2.1
fastrlock 0.8.3
fastuuid 0.13.5
ffmpy 0.6.1
filelock 3.19.1
fire 0.7.1
flash_attn 2.7.4.post1
flashinfer-python 0.2.9rc2
fonttools 4.60.1
frozenlist 1.7.0
fsspec 2025.3.0
funasr 1.2.7
func-timeout 4.3.5
future 1.0.0
fuzzywuzzy 0.18.0
gekko 1.3.0
genson 1.3.0
gguf 0.17.1
gitdb 4.0.12
GitPython 3.1.45
google-auth 2.41.1
google-genai 1.39.1
gradio 5.32.1
gradio_client 1.10.2
groovy 0.1.2
grpcio 1.75.1
h11 0.16.0
h5py 3.14.0
hf_transfer 0.1.9
hf-xet 1.1.10
hjson 3.1.0
httpcore 1.0.9
httptools 0.6.4
httpx 0.28.1
huggingface-hub 0.35.3
human-eval 1.0.3
hydra-core 1.3.2
icecream 2.1.6
identify 2.6.14
idna 3.10
imageio 2.37.0
immutabledict 4.2.1
importlib_metadata 8.7.0
interegular 0.3.3
ipdb 0.13.13
ipykernel 7.0.1
ipython 9.6.0
ipython_pygments_lexers 1.1.1
jaconv 0.4.0
jamo 0.4.1
jedi 0.19.2
jieba 0.42.1
Jinja2 3.1.6
jiter 0.11.0
jmespath 0.10.0
joblib 1.5.2
json_repair 0.51.0
json5 0.12.1
jsonlines 4.0.0
jsonpath-ng 1.7.0
jsonschema 4.25.1
jsonschema-specifications 2025.9.1
jupyter_client 8.6.3
jupyter_core 5.8.1
kaldiio 2.18.1
kiwisolver 1.4.9
langdetect 1.0.9
lark 1.2.2
latex2sympy2_extended 1.0.6
lazy_loader 0.4
Levenshtein 0.27.1
librosa 0.11.0
liger_kernel 0.6.2
lightning-utilities 0.15.2
litellm 1.77.5
llguidance 0.7.30
llvmlite 0.44.0
lm-format-enforcer 0.10.12
lmdeploy 0.10.1
lxml 6.0.2
Markdown 3.9
markdown-it-py 4.0.0
MarkupSafe 3.0.3
math-verify 0.5.2
matplotlib 3.10.6
matplotlib-inline 0.1.7
mdurl 0.1.2
megatron-core 0.13.2
mistral_common 1.8.5
ml_dtypes 0.5.3
mmengine-lite 0.10.7
modelscope 1.29.2
more-itertools 10.8.0
mpmath 1.3.0
ms-agent 1.3.0
ms-opencompass 0.1.6
ms-swift 3.10.0.dev0 /xxx/ms-swift-3.9.0
ms-vlmeval 0.0.18
msgpack 1.1.1
msgspec 0.19.0
multidict 6.6.4
multiprocess 0.70.16
nest-asyncio 1.6.0
networkx 3.5
ninja 1.13.0
nltk 3.9.2
nodeenv 1.9.1
numba 0.61.2
numpy 1.26.4
nvidia-cublas-cu12 12.6.4.1
nvidia-cuda-cupti-cu12 12.6.80
nvidia-cuda-nvrtc-cu12 12.6.77
nvidia-cuda-runtime-cu12 12.6.77
nvidia-cudnn-cu12 9.5.1.17
nvidia-cudnn-frontend 1.14.1
nvidia-cufft-cu12 11.3.0.4
nvidia-cufile-cu12 1.11.1.6
nvidia-curand-cu12 10.3.7.77
nvidia-cusolver-cu12 11.7.1.2
nvidia-cusparse-cu12 12.5.4.2
nvidia-cusparselt-cu12 0.6.3
nvidia-ml-py 13.580.82
nvidia-nccl-cu12 2.26.2
nvidia-nvjitlink-cu12 12.6.85
nvidia-nvshmem-cu12 3.4.5
nvidia-nvtx-cu12 12.6.77
nvitop 1.5.3
omegaconf 2.3.0
onnx 1.19.0
onnx-ir 0.1.10
onnxscript 0.3.1
openai 2.0.0
openai-harmony 0.0.4
openai-whisper 20250625
OpenCC 1.1.9
opencv-python 4.11.0.86
opencv-python-headless 4.12.0.88
openpyxl 3.1.5
optimum 1.27.0
orjson 3.11.3
oss2 2.19.1
outlines 0.1.11
outlines_core 0.1.26
overrides 7.7.0
packaging 25.0
pandas 2.3.2
parso 0.8.5
partial-json-parser 0.2.1.1.post6
peft 0.14.0
pexpect 4.9.0
pillow 11.3.0
pip 23.3.2
platformdirs 4.4.0
ply 3.11
pooch 1.8.2
portalocker 3.2.0
pre_commit 4.3.0
prettytable 3.16.0
prometheus_client 0.23.1
prometheus-fastapi-instrumentator 7.1.0
prompt_toolkit 3.0.52
propcache 0.3.2
protobuf 6.32.1
psutil 7.1.0
ptyprocess 0.7.0
pure_eval 0.2.3
py-cpuinfo 9.0.0
py-spy 0.4.1
pyarrow 21.0.0
pyasn1 0.6.1
pyasn1_modules 0.4.2
pybase64 1.4.2
pybind11 3.0.0
pycountry 24.6.1
pycparser 2.22
pycryptodome 3.23.0
pycryptodomex 3.23.0
pydantic 2.11.9
pydantic_core 2.33.2
pydantic-extra-types 2.10.5
pydub 0.25.1
pyecharts 2.0.8
Pygments 2.19.2
pynndescent 0.5.13
pynvml 13.0.1
pyparsing 3.2.5
pypinyin 0.55.0
pyproject_hooks 1.2.0
python-dateutil 2.9.0.post0
python-dotenv 1.1.1
python-json-logger 3.3.0
python-Levenshtein 0.27.1
python-multipart 0.0.20
pytorch-wpe 0.0.1
pytz 2025.2
PyYAML 6.0.2
pyzmq 27.1.0
qwen-omni-utils 0.0.8
qwen-vl-utils 0.0.14
rank-bm25 0.2.2
RapidFuzz 3.14.1
ray 2.49.2
referencing 0.36.2
regex 2025.9.1
requests 2.32.5
rich 13.9.4
rich-toolkit 0.15.1
rignore 0.6.4
rouge 1.0.1
rouge-chinese 1.0.3
rouge-score 0.1.2
rpds-py 0.27.1
rsa 4.9.1
ruff 0.13.2
s3transfer 0.14.0
sacrebleu 2.5.1
safehttpx 0.1.6
safetensors 0.6.2
scikit-learn 1.7.2
scipy 1.16.2
seaborn 0.13.2
semantic-version 2.10.0
sentence-transformers 5.1.1
sentencepiece 0.2.1
sentry-sdk 2.39.0
setproctitle 1.3.7
setuptools 65.5.1
sgl-kernel 0.2.8
sglang 0.4.10.post2
shellingham 1.5.4
shortuuid 1.0.13
simplejson 3.20.1
six 1.17.0
smmap 5.0.2
sniffio 1.3.1
sortedcontainers 2.4.0
soundfile 0.13.1
soxr 1.0.0
stack-data 0.6.3
starlette 0.48.0
sty 1.0.6
swankit 0.2.4
swanlab 0.6.10
sympy 1.14.0
tabulate 0.9.0
tenacity 9.1.2
tensorboard 2.20.0
tensorboard-data-server 0.7.2
tensorboardX 2.6.4
termcolor 3.1.0
threadpoolctl 3.6.0
tiktoken 0.11.0
timeout-decorator 0.5.0
timm 1.0.16
tokenizers 0.22.1
tomlkit 0.13.3
torch 2.7.1
torch-complex 0.4.4
torch_memory_saver 0.0.8
torchao 0.9.0
torchaudio 2.7.1
torchmetrics 1.8.2
torchvision 0.22.1
tornado 6.5.2
tqdm 4.67.1
traitlets 5.14.3
transformer_engine 2.6.0.post1
transformer_engine_cu12 2.6.0.post1
transformer-engine-torch 2.6.0.post1
transformers 4.57.1
transformers-stream-generator 0.0.5
triton 3.3.1
trl 0.20.0
typer 0.19.2
typing_extensions 4.15.0
typing-inspection 0.4.2
tzdata 2025.2
umap-learn 0.5.9.post2
urllib3 2.5.0
uvicorn 0.37.0
uvloop 0.21.0
validators 0.35.0
virtualenv 20.34.0
vllm 0.10.1.1
vlmeval 0.1.0 /xxx/VLMEvalKit-1019
wandb 0.22.1
watchfiles 1.1.0
wcwidth 0.2.14
websockets 15.0.1
Werkzeug 3.1.3
wheel 0.45.1
word2number 1.1
wrapt 1.17.3
xformers 0.0.31
xgrammar 0.1.22
xlsxwriter 3.2.9
xxhash 3.5.0
yapf 0.43.0
yarl 1.20.1
zipp 3.23.0
zstandard 0.25.0`

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided lmdeploy serve api_server Qwen/Qwen3-VL-4B-Instruct --server-port 23333 command and locate VLMEvalKit’s lmdeploy integration; no repository file or test is named in the issue. Done means Qwen3-VL-4B-Instruct is supported and its serving or evaluation path works, with project-specific validation identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.