open-compass / open-compass/VLMEvalKit

3DSRBench和SiteBenchImage两个数据集评测的bug麻烦修复下,修复方式文中给出。

Open
#1,485 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.4k
Forks
768
Avg merge
1d 10h
Merged PRs (30d)
17

Description

评测过程中发现两个bug,麻烦有空帮忙修正下:
1.3DSRBench评测name_str未定义。
vlmeval/dataset/image_mcq.py文件中2576行的name_str应该得和其他的一样加上类似下面的代码

        name_str_map = {'chatgpt-0125': 'openai', 'gpt-4-0125': 'gpt4'}
        name_str = name_str_map[model] if model in name_str_map else model

2.SiteBenchImage读图时会有没扩展名的文件。
vlmeval/smp/file.py文件378行的parse_file函数应该参考https://github.com/EvolvingLMMs-Lab/VLMEvalKit/blob/EASI/vlmeval/smp/file.py
修改为如下

def _mime_from_pillow(p):
    from PIL import Image
    try:
        with Image.open(p) as im:
            fmt = (im.format or '').lower()
        return {
            'jpeg': 'image/jpeg', 'jpg': 'image/jpeg',
            'png': 'image/png',
            'webp': 'image/webp',
            'bmp': 'image/bmp',
            'gif': 'image/gif',
            'tiff': 'image/tiff', 'tif': 'image/tiff',
        }.get(fmt, 'unknown')
    except Exception:
        return 'unknown'


def parse_file(s):
    if osp.exists(s) and s != '.':
        assert osp.isfile(s)
        suffix = osp.splitext(s)[1].lower()
        # 添加对webp的支持
        if suffix == '.webp':
            return ('image/webp', s)
        mime = mimetypes.types_map.get(suffix, 'unknown')
        if mime == 'unknown':
            mime = _mime_from_pillow(s)
        return (mime, s)
    elif s.startswith('data:image/'):
        # To be compatible with OPENAI base64 format
        content = s[11:]
        mime = content.split(';')[0]
        content = ';'.join(content.split(';')[1:])
        dname = osp.join(LMUDataRoot(), 'files')
        assert content.startswith('base64,')
        b64 = content[7:]
        os.makedirs(dname, exist_ok=True)
        tgt = osp.join(dname, md5(b64) + '.png')
        decode_base64_to_image_file(b64, tgt)
        return parse_file(tgt)
    elif validators.url(s):
        suffix = osp.splitext(s)[1].lower()
        # 添加对webp的支持
        if suffix == '.webp':
            mime = 'image/webp'
        elif suffix in mimetypes.types_map:
            mime = mimetypes.types_map[suffix]
            dname = osp.join(LMUDataRoot(), 'files')
            os.makedirs(dname, exist_ok=True)
            tgt = osp.join(dname, md5(s) + suffix)
            download_file(s, tgt)
            return (mime, tgt)
        else:
            return ('url', s)

    else:
        return (None, s)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with vlmeval/dataset/image_mcq.py around line 2576 and vlmeval/smp/file.py around parse_file at line 378. Compare the latter with the linked EASI version, then evaluate 3DSRBench and SiteBenchImage to confirm model names resolve and extensionless image files are read successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.