open-compass / open-compass/opencompass
[Bug] 多卡测试llama-3-8b-vllm,精度为0
@acylam is already working on this.
Since Apr 8, 2025.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 869
- Avg merge
- 17h 52m
- Merged PRs (30d)
- 13
Description
Prerequisite
- I have searched Issues and Discussions but cannot get the expected help.
- The bug has not been fixed in the latest version.
Type
I'm evaluating with the officially supported tasks/models/datasets.
Environment
python run.py configs/vllm/eval_llama3_vllm.py
Reproduces the problem - code/configuration sample
from mmengine.config import read_base
with read_base():
from ..datasets.ARC_c.ARC_c_gen_1e0de5 import ARC_c_datasets
#from ..datasets.ARC_e.ARC_e_gen_1e0de5 import ARC_e_datasets
from ..summarizers.example import summarizer
datasets = sum([v for k, v in locals().items() if k.endswith("_datasets") or k == 'datasets'], [])
work_dir = './outputs/llama3/'
from opencompass.models import VLLM
models = [
dict(
type=VLLM,
abbr='llama-3-8b-vllm',
path="./models/Meta-Llama-3-8B",
model_kwargs=dict(tensor_parallel_size=4),
max_out_len=100,
max_seq_len=2048,
batch_size=16,
generation_kwargs=dict(temperature=0),
run_cfg=dict(num_gpus=4, num_procs=1),
)
]
Reproduces the problem - command or script
四张卡测试ARC精度,精度为0
Reproduces the problem - error message
Other information
No response
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.