open-compass / open-compass/opencompass

[Bug] 多卡测试llama-3-8b-vllm,精度为0

Open
#1,979 2 comments 0 reactions 1 assignee View on GitHub

@acylam is already working on this.

Since Apr 8, 2025.

Dominant language
Python
Stars
7.5k
Forks
869
Avg merge
17h 52m
Merged PRs (30d)
13

Description

Prerequisite
Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

python run.py configs/vllm/eval_llama3_vllm.py

Reproduces the problem - code/configuration sample

from mmengine.config import read_base

with read_base():
from ..datasets.ARC_c.ARC_c_gen_1e0de5 import ARC_c_datasets
#from ..datasets.ARC_e.ARC_e_gen_1e0de5 import ARC_e_datasets
from ..summarizers.example import summarizer

datasets = sum([v for k, v in locals().items() if k.endswith("_datasets") or k == 'datasets'], [])
work_dir = './outputs/llama3/'

from opencompass.models import VLLM

models = [
dict(
type=VLLM,
abbr='llama-3-8b-vllm',
path="./models/Meta-Llama-3-8B",
model_kwargs=dict(tensor_parallel_size=4),
max_out_len=100,
max_seq_len=2048,
batch_size=16,
generation_kwargs=dict(temperature=0),
run_cfg=dict(num_gpus=4, num_procs=1),
)
]

Reproduces the problem - command or script

四张卡测试ARC精度,精度为0

Reproduces the problem - error message

Image

Other information

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.