open-compass / open-compass/VLMEvalKit

Dataset GMAI-MMBench_VAL is not working

Open
#1,173 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.4k
Forks
768
Avg merge
2d 27m
Merged PRs (30d)
18

Description

!python run.py --data GMAI-MMBench_VAL --model idefics_9b_instruct --verbose. The result is [2025-07-21 11:59:53] ERROR - misc.py: load_env - 215: Did not detect the .env file at /content/VLMEvalKit/.env, failed to load.
2025-07-21 11:59:57.664789: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1753099197.685422 19582 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1753099197.691648 19582 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
[2025-07-21 12:00:01] ERROR - misc.py: load_env - 215: Did not detect the .env file at /content/VLMEvalKit/.env, failed to load.
[2025-07-21 12:00:01] WARNING - RUN - run.py: main - 216: --reuse is not set, will not reuse previous (before one day) temporary files
[2025-07-21 12:00:01] ERROR - RUN - run.py: main - 472: Model idefics_9b_instruct x Dataset GMAI-MMBench_VAL combination failed: [Errno 2] No such file or directory: '/root/LMUData/GMAI-MMBench_VAL.tsv', skipping this combination.
Traceback (most recent call last):
File "/content/VLMEvalKit/run.py", line 295, in main
dataset = build_dataset(dataset_name, **dataset_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/VLMEvalKit/vlmeval/dataset/init.py", line 279, in build_dataset
return cls(dataset=dataset_name, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/VLMEvalKit/vlmeval/dataset/image_base.py", line 45, in init
data = self.load_data(dataset)
^^^^^^^^^^^^^^^^^^^^^^^
File "/content/VLMEvalKit/vlmeval/dataset/image_mcq.py", line 630, in load_data
if file_size(data_path, 'GB') > 1:
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/VLMEvalKit/vlmeval/smp/file.py", line 346, in file_size
stats = os.stat(f)
^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/root/LMUData/GMAI-MMBench_VAL.tsv'
Can any one help?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the command in the issue and inspect run.py's dataset setup. Follow the traceback through vlmeval/dataset/image_base.py, image_mcq.py, and smp/file.py, then verify how GMAI-MMBench_VAL is expected to provide /root/LMUData/GMAI-MMBench_VAL.tsv. Done means the dataset loads and the model-dataset evaluation runs without this FileNotFoundError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.