open-compass / open-compass/VLMEvalKit

Qwen2.5vl 7b can't reproduce Mathvista_MINI like leaderboard

Open
#874 2 comments 1 reaction 1 assignee View on GitHub

@FangXinyu-0913 is already working on this.

Since Mar 27, 2025.

Dominant language
Python
Stars
4.4k
Forks
768
Avg merge
1d 10h
Merged PRs (30d)
17

Description

Results(66.6 not align with leaderboard 68.1):
Image
CUDA:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.04 Driver Version: 570.124.04 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100 80GB PCIe Off | 00000001:00:00.0 Off | 0 |
| N/A 33C P0 41W / 300W | 6MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
envs:
transformers:4.49.0
torch:2.5.1
torchvision:0.20.0

Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
accelerate 1.5.2 pypi_0 pypi
aiofiles 23.2.1 pypi_0 pypi
annotated-types 0.7.0 pypi_0 pypi
antlr4-python3-runtime 4.9.3 pypi_0 pypi
anyio 4.9.0 pypi_0 pypi
av 14.2.0 pypi_0 pypi
bzip2 1.0.8 h5eee18b_6
ca-certificates 2025.2.25 h06a4308_0
certifi 2025.1.31 pypi_0 pypi
charset-normalizer 3.4.1 pypi_0 pypi
click 8.1.8 pypi_0 pypi
contourpy 1.3.1 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
decord 0.6.0 pypi_0 pypi
distro 1.9.0 pypi_0 pypi
einops 0.8.1 pypi_0 pypi
et-xmlfile 2.0.0 pypi_0 pypi
exceptiongroup 1.2.2 pypi_0 pypi
fastapi 0.115.12 pypi_0 pypi
ffmpy 0.5.0 pypi_0 pypi
filelock 3.18.0 pypi_0 pypi
flash-attn 2.7.4.post1 pypi_0 pypi
fonttools 4.56.0 pypi_0 pypi
fsspec 2025.3.0 pypi_0 pypi
gradio 5.23.0 pypi_0 pypi
gradio-client 1.8.0 pypi_0 pypi
groovy 0.1.2 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
httpcore 1.0.7 pypi_0 pypi
httpx 0.28.1 pypi_0 pypi
huggingface-hub 0.29.3 pypi_0 pypi
idna 3.10 pypi_0 pypi
imageio 2.37.0 pypi_0 pypi
jinja2 3.1.6 pypi_0 pypi
jiter 0.9.0 pypi_0 pypi
kiwisolver 1.4.8 pypi_0 pypi
ld_impl_linux-64 2.40 h12ee557_0
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
libuuid 1.41.5 h5eee18b_0
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 3.0.2 pypi_0 pypi
matplotlib 3.10.1 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
ncurses 6.4 h6a678d5_0
networkx 3.4.2 pypi_0 pypi
numpy 2.2.4 pypi_0 pypi
nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi
nvidia-curand-cu12 10.3.5.147 pypi_0 pypi
nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi
nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi
nvidia-cusparselt-cu12 0.6.2 pypi_0 pypi
nvidia-nccl-cu12 2.21.5 pypi_0 pypi
nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
nvidia-nvtx-cu12 12.4.127 pypi_0 pypi
omegaconf 2.3.0 pypi_0 pypi
openai 1.68.2 pypi_0 pypi
opencv-python 4.11.0.86 pypi_0 pypi
openpyxl 3.1.5 pypi_0 pypi
openssl 1.1.1w h7f8727e_0
orjson 3.10.16 pypi_0 pypi
packaging 24.2 pypi_0 pypi
pandas 2.2.3 pypi_0 pypi
pillow 11.1.0 pypi_0 pypi
pip 25.0 py310h06a4308_0
portalocker 3.1.1 pypi_0 pypi
protobuf 6.30.1 pypi_0 pypi
psutil 7.0.0 pypi_0 pypi
pydantic 2.10.6 pypi_0 pypi
pydantic-core 2.27.2 pypi_0 pypi
pydub 0.25.1 pypi_0 pypi
pygments 2.19.1 pypi_0 pypi
pyparsing 3.2.3 pypi_0 pypi
python 3.10.0 h12debd9_5
python-dateutil 2.9.0.post0 pypi_0 pypi
python-dotenv 1.0.1 pypi_0 pypi
python-multipart 0.0.20 pypi_0 pypi
pytz 2025.2 pypi_0 pypi
pyyaml 6.0.2 pypi_0 pypi
qwen-vl-utils 0.0.10 pypi_0 pypi
readline 8.2 h5eee18b_0
regex 2024.11.6 pypi_0 pypi
requests 2.32.3 pypi_0 pypi
rich 13.9.4 pypi_0 pypi
ruff 0.11.2 pypi_0 pypi
safehttpx 0.1.6 pypi_0 pypi
safetensors 0.5.3 pypi_0 pypi
semantic-version 2.10.0 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
setuptools 75.8.0 py310h06a4308_0
shellingham 1.5.4 pypi_0 pypi
six 1.17.0 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
sqlite 3.45.3 h5eee18b_0
starlette 0.46.1 pypi_0 pypi
sty 1.0.6 pypi_0 pypi
sympy 1.13.1 pypi_0 pypi
tabulate 0.9.0 pypi_0 pypi
tiktoken 0.9.0 pypi_0 pypi
timeout-decorator 0.5.0 pypi_0 pypi
tk 8.6.14 h39e8969_0
tokenizers 0.21.1 pypi_0 pypi
tomlkit 0.13.2 pypi_0 pypi
torch 2.5.1 pypi_0 pypi
torchvision 0.20.0 pypi_0 pypi
tqdm 4.67.1 pypi_0 pypi
transformers 4.49.0 pypi_0 pypi
triton 3.1.0 pypi_0 pypi
typer 0.15.2 pypi_0 pypi
typing-extensions 4.12.2 pypi_0 pypi
tzdata 2025.2 pypi_0 pypi
urllib3 2.3.0 pypi_0 pypi
uv 0.6.9 pypi_0 pypi
uvicorn 0.34.0 pypi_0 pypi
validators 0.34.0 pypi_0 pypi
vlmeval 0.1.0 dev_0
websockets 15.0.1 pypi_0 pypi
wheel 0.45.1 py310h06a4308_0
xlsxwriter 3.2.2 pypi_0 pypi
xz 5.6.4 h5eee18b_1
zlib 1.2.13 h5eee18b_1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.