MMT-Bench Assertion Error

Command:

python run.py --data MMT-Bench_ALL_MI --model llava_next_interleave_7b --verbose

Output:

[2025-01-04 17:01:05] ERROR - misc.py: load_env - 212: Did not detect the .env file at /home/mengyu/VLMEvalKit/.env, failed to load. 
[2025-01-04 17:01:05] ERROR - misc.py: load_env - 212: Did not detect the .env file at /home/mengyu/VLMEvalKit/.env, failed to load. 
[2025-01-04 17:01:05,704] WARNING - RUN - run.py: main - 165: --reuse is not set, will not reuse previous (before one day) temporary files
[2025-01-04 17:01:05] WARNING - run.py: main - 165: --reuse is not set, will not reuse previous (before one day) temporary files
[2025-01-04 17:02:32,041] ERROR - RUN - run.py: main - 411: Model llava_next_interleave_7b x Dataset MMT-Bench_ALL_MI combination failed: , skipping this combination.
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/mengyu/VLMEvalKit/vlmeval/smp/file.py", line 24, in decode_img_omni
    assert len(images) == len(paths)
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mengyu/VLMEvalKit/run.py", line 232, in main
    dataset = build_dataset(dataset_name, **dataset_kwargs)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/__init__.py", line 207, in build_dataset
    return cls(dataset=dataset_name, **kwargs)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/image_base.py", line 45, in __init__
    data = self.load_data(dataset)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/image_base.py", line 143, in load_data
    return self.prepare_tsv(url, file_md5)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/image_base.py", line 101, in prepare_tsv
    LOCALIZE(data_path, local_path)
  File "/home/mengyu/VLMEvalKit/vlmeval/tools.py", line 291, in LOCALIZE
    data_new = localize_df(data, dname)
  File "/home/mengyu/VLMEvalKit/vlmeval/smp/file.py", line 60, in localize_df
    ret = pool.map(decode_img_omni, tups)
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
AssertionError
[2025-01-04 17:02:32] ERROR - run.py: main - 411: Model llava_next_interleave_7b x Dataset MMT-Bench_ALL_MI combination failed: , skipping this combination.
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/mengyu/VLMEvalKit/vlmeval/smp/file.py", line 24, in decode_img_omni
    assert len(images) == len(paths)
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mengyu/VLMEvalKit/run.py", line 232, in main
    dataset = build_dataset(dataset_name, **dataset_kwargs)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/__init__.py", line 207, in build_dataset
    return cls(dataset=dataset_name, **kwargs)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/image_base.py", line 45, in __init__
    data = self.load_data(dataset)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/image_base.py", line 143, in load_data
    return self.prepare_tsv(url, file_md5)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/image_base.py", line 101, in prepare_tsv
    LOCALIZE(data_path, local_path)
  File "/home/mengyu/VLMEvalKit/vlmeval/tools.py", line 291, in LOCALIZE
    data_new = localize_df(data, dname)
  File "/home/mengyu/VLMEvalKit/vlmeval/smp/file.py", line 60, in localize_df
    ret = pool.map(decode_img_omni, tups)
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
AssertionError

The dataset is automatically downloaded by the program. How can I fix this problem?

Start with the reported command in run.py and trace dataset preparation through vlmeval/dataset/image_base.py, vlmeval/tools.py, and vlmeval/smp/file.py. Inspect decode_img_omni and the MMT-Bench_ALL_MI data passed to it, focusing on the failing image/path length assertion. Done means the command can localize the automatically downloaded dataset without raising AssertionError.