open-compass / open-compass/VLMEvalKit
KeyError: 'answer'
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
File "/root/.local/lib/python3.12/site-packages/vlmeval/dataset/image_mcq.py", line 181, in evaluate
answer_map = {i: c for i, c in zip(meta['index'], meta['answer'])}
~~~~^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/pandas/core/frame.py", line 4090, in getitem
indexer = self.columns.get_loc(key)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
raise KeyError(key) from err
KeyError: 'answer'
Hello, how to resolve this issue when eval on MME
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with vlmeval/dataset/image_mcq.py around line 181 and reproduce evaluation on MME. Inspect the metadata columns available at the failure and trace how the MME input reaches evaluate. Done means MME evaluation completes without the KeyError and produces its expected results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100