open-compass / open-compass/VLMEvalKit
Dataset GMAI-MMBench_VAL is not working
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 2d 27m
- Merged PRs (30d)
- 18
Description
!python run.py --data GMAI-MMBench_VAL --model idefics_9b_instruct --verbose. The result is [2025-07-21 11:59:53] ERROR - misc.py: load_env - 215: Did not detect the .env file at /content/VLMEvalKit/.env, failed to load.
2025-07-21 11:59:57.664789: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1753099197.685422 19582 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1753099197.691648 19582 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
[2025-07-21 12:00:01] ERROR - misc.py: load_env - 215: Did not detect the .env file at /content/VLMEvalKit/.env, failed to load.
[2025-07-21 12:00:01] WARNING - RUN - run.py: main - 216: --reuse is not set, will not reuse previous (before one day) temporary files
[2025-07-21 12:00:01] ERROR - RUN - run.py: main - 472: Model idefics_9b_instruct x Dataset GMAI-MMBench_VAL combination failed: [Errno 2] No such file or directory: '/root/LMUData/GMAI-MMBench_VAL.tsv', skipping this combination.
Traceback (most recent call last):
File "/content/VLMEvalKit/run.py", line 295, in main
dataset = build_dataset(dataset_name, **dataset_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/VLMEvalKit/vlmeval/dataset/init.py", line 279, in build_dataset
return cls(dataset=dataset_name, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/VLMEvalKit/vlmeval/dataset/image_base.py", line 45, in init
data = self.load_data(dataset)
^^^^^^^^^^^^^^^^^^^^^^^
File "/content/VLMEvalKit/vlmeval/dataset/image_mcq.py", line 630, in load_data
if file_size(data_path, 'GB') > 1:
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/content/VLMEvalKit/vlmeval/smp/file.py", line 346, in file_size
stats = os.stat(f)
^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/root/LMUData/GMAI-MMBench_VAL.tsv'
Can any one help?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the command in the issue and inspect run.py's dataset setup. Follow the traceback through vlmeval/dataset/image_base.py, image_mcq.py, and smp/file.py, then verify how GMAI-MMBench_VAL is expected to provide /root/LMUData/GMAI-MMBench_VAL.tsv. Done means the dataset loads and the model-dataset evaluation runs without this FileNotFoundError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100