open-compass / open-compass/VLMEvalKit
The evaluation device_map setting is not optimized for quantized models using the ExLlamaV2 backend.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
The evaluation setting is not optimized for quantized models using the ExLlamaV2 backend, which serves as the default kernel for Auto-GPTQ and Transformers.
Changing device_map="cpu" to device_map='auto' will resolve the model loading issue.
The known problematic model files are:
vlmeval/vlm/llama_vision.py
vlmeval/vlm/llava/llava.py
vlmeval/vlm/qwen2_vl/model.py
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting vlmeval/vlm/llama_vision.py, vlmeval/vlm/llava/llava.py, and vlmeval/vlm/qwen2_vl/model.py, focusing on the evaluation device_map setting. Replace the problematic CPU mapping with the reported automatic mapping, then verify that the affected quantized models load successfully through the evaluation workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100