open-compass / open-compass/VLMEvalKit

Why doesn’t VizWiz’s `build_prompt` include the short-answer instruction like `ImageVQADataset`?

Open
#1,090 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.4k
Forks
768
Avg merge
1d 10h
Merged PRs (30d)
17

Description

I noticed that in ImageVQADataset you override build_prompt to append:

Answer the question using a single word or phrase.

This ensures the model knows to produce a short answer. However, the VizWiz class inherits build_prompt directly from ImageBaseDataset and doesn’t include this instruction, which leads to long descriptive outputs that always score 0 under VQA Accuracy.

Could you clarify why this short-answer prompt is added for ImageVQADataset but not for VizWiz? Is there a reason for treating VizWiz differently, or would you consider adding the same instruction to improve scoring?

Empirical Observation:
After manually adding this line to VizWiz.build_prompt, Ovis2-8B’s VQA Accuracy jumped from 3.09% to 39.77% on the VizWiz evaluation set.

Thank you!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Compare ImageVQADataset.build_prompt with the inherited ImageBaseDataset implementation used by VizWiz. Check the VizWiz evaluation set and VQA Accuracy behavior after confirming the intended prompt format. Done means the project has a documented or implemented decision for whether VizWiz should receive the short-answer instruction, with evaluation results supporting it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, testing
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.