open-compass / open-compass/VLMEvalKit
如何使用本地测试数据集
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.4k
- Forks
- 768
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 17
Description
因为我们的服务器无法直接连接到huggenface，所以我们下载MMBench-Video数据集到本地然后放置到/home/amsr/LMUData目录下。然后使用vllm启动api，如下所示：
python3 -m vllm.entrypoints.openai.api_server --model "~/Qwen2-VL-7B-Instruct" --dtype=half
运行VLMEvalKit：
python3 run.py --data MMBench-Video --model lmdeploy --verbose --api-nproc 16
报错如下：An error happened while trying to locate the files on the Hub and we cannot find the appropriate snapshot folder for the specified revision on the local disk. Please check your internet connection and try again.
可否提供一个使用vllm运行本地数据集进行评测的例子？
谢谢
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with run.py and the documented MMBench-Video invocation, then check how the local /home/amsr/LMUData dataset and the vllm API endpoint are resolved. Document a reproducible command sequence using the local dataset and Qwen2-VL-7B-Instruct server, with completion shown by an evaluation run that no longer tries to locate files on the Hub.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100