open-compass / open-compass/VLMEvalKit
Qwen2.5vl 7b can't reproduce Mathvista_MINI like leaderboard
@FangXinyu-0913 is already working on this.
Since Mar 27, 2025.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
Results(66.6 not align with leaderboard 68.1):
CUDA:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.04 Driver Version: 570.124.04 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100 80GB PCIe Off | 00000001:00:00.0 Off | 0 |
| N/A 33C P0 41W / 300W | 6MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
envs:
transformers:4.49.0
torch:2.5.1
torchvision:0.20.0
Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
accelerate 1.5.2 pypi_0 pypi
aiofiles 23.2.1 pypi_0 pypi
annotated-types 0.7.0 pypi_0 pypi
antlr4-python3-runtime 4.9.3 pypi_0 pypi
anyio 4.9.0 pypi_0 pypi
av 14.2.0 pypi_0 pypi
bzip2 1.0.8 h5eee18b_6
ca-certificates 2025.2.25 h06a4308_0
certifi 2025.1.31 pypi_0 pypi
charset-normalizer 3.4.1 pypi_0 pypi
click 8.1.8 pypi_0 pypi
contourpy 1.3.1 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
decord 0.6.0 pypi_0 pypi
distro 1.9.0 pypi_0 pypi
einops 0.8.1 pypi_0 pypi
et-xmlfile 2.0.0 pypi_0 pypi
exceptiongroup 1.2.2 pypi_0 pypi
fastapi 0.115.12 pypi_0 pypi
ffmpy 0.5.0 pypi_0 pypi
filelock 3.18.0 pypi_0 pypi
flash-attn 2.7.4.post1 pypi_0 pypi
fonttools 4.56.0 pypi_0 pypi
fsspec 2025.3.0 pypi_0 pypi
gradio 5.23.0 pypi_0 pypi
gradio-client 1.8.0 pypi_0 pypi
groovy 0.1.2 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
httpcore 1.0.7 pypi_0 pypi
httpx 0.28.1 pypi_0 pypi
huggingface-hub 0.29.3 pypi_0 pypi
idna 3.10 pypi_0 pypi
imageio 2.37.0 pypi_0 pypi
jinja2 3.1.6 pypi_0 pypi
jiter 0.9.0 pypi_0 pypi
kiwisolver 1.4.8 pypi_0 pypi
ld_impl_linux-64 2.40 h12ee557_0
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
libuuid 1.41.5 h5eee18b_0
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 3.0.2 pypi_0 pypi
matplotlib 3.10.1 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
ncurses 6.4 h6a678d5_0
networkx 3.4.2 pypi_0 pypi
numpy 2.2.4 pypi_0 pypi
nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi
nvidia-curand-cu12 10.3.5.147 pypi_0 pypi
nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi
nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi
nvidia-cusparselt-cu12 0.6.2 pypi_0 pypi
nvidia-nccl-cu12 2.21.5 pypi_0 pypi
nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
nvidia-nvtx-cu12 12.4.127 pypi_0 pypi
omegaconf 2.3.0 pypi_0 pypi
openai 1.68.2 pypi_0 pypi
opencv-python 4.11.0.86 pypi_0 pypi
openpyxl 3.1.5 pypi_0 pypi
openssl 1.1.1w h7f8727e_0
orjson 3.10.16 pypi_0 pypi
packaging 24.2 pypi_0 pypi
pandas 2.2.3 pypi_0 pypi
pillow 11.1.0 pypi_0 pypi
pip 25.0 py310h06a4308_0
portalocker 3.1.1 pypi_0 pypi
protobuf 6.30.1 pypi_0 pypi
psutil 7.0.0 pypi_0 pypi
pydantic 2.10.6 pypi_0 pypi
pydantic-core 2.27.2 pypi_0 pypi
pydub 0.25.1 pypi_0 pypi
pygments 2.19.1 pypi_0 pypi
pyparsing 3.2.3 pypi_0 pypi
python 3.10.0 h12debd9_5
python-dateutil 2.9.0.post0 pypi_0 pypi
python-dotenv 1.0.1 pypi_0 pypi
python-multipart 0.0.20 pypi_0 pypi
pytz 2025.2 pypi_0 pypi
pyyaml 6.0.2 pypi_0 pypi
qwen-vl-utils 0.0.10 pypi_0 pypi
readline 8.2 h5eee18b_0
regex 2024.11.6 pypi_0 pypi
requests 2.32.3 pypi_0 pypi
rich 13.9.4 pypi_0 pypi
ruff 0.11.2 pypi_0 pypi
safehttpx 0.1.6 pypi_0 pypi
safetensors 0.5.3 pypi_0 pypi
semantic-version 2.10.0 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
setuptools 75.8.0 py310h06a4308_0
shellingham 1.5.4 pypi_0 pypi
six 1.17.0 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
sqlite 3.45.3 h5eee18b_0
starlette 0.46.1 pypi_0 pypi
sty 1.0.6 pypi_0 pypi
sympy 1.13.1 pypi_0 pypi
tabulate 0.9.0 pypi_0 pypi
tiktoken 0.9.0 pypi_0 pypi
timeout-decorator 0.5.0 pypi_0 pypi
tk 8.6.14 h39e8969_0
tokenizers 0.21.1 pypi_0 pypi
tomlkit 0.13.2 pypi_0 pypi
torch 2.5.1 pypi_0 pypi
torchvision 0.20.0 pypi_0 pypi
tqdm 4.67.1 pypi_0 pypi
transformers 4.49.0 pypi_0 pypi
triton 3.1.0 pypi_0 pypi
typer 0.15.2 pypi_0 pypi
typing-extensions 4.12.2 pypi_0 pypi
tzdata 2025.2 pypi_0 pypi
urllib3 2.3.0 pypi_0 pypi
uv 0.6.9 pypi_0 pypi
uvicorn 0.34.0 pypi_0 pypi
validators 0.34.0 pypi_0 pypi
vlmeval 0.1.0 dev_0
websockets 15.0.1 pypi_0 pypi
wheel 0.45.1 py310h06a4308_0
xlsxwriter 3.2.2 pypi_0 pypi
xz 5.6.4 h5eee18b_1
zlib 1.2.13 h5eee18b_1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.