open-compass / open-compass/opencompass

[Bug] 根据官方的Quick Start运行,输出结果为空

Open
#1,168 17 comments 0 reactions 1 assignee View on GitHub

@kennymckormick is already working on this.

Since May 16, 2024.

Dominant language
Python
Stars
7.5k
Forks
869
Avg merge
17h 52m
Merged PRs (30d)
13

Description

Prerequisite
Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

{'CUDA available': True,
'CUDA_HOME': None,
'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0',
'GPU 0': 'NVIDIA GeForce RTX 3090',
'MMEngine': '0.10.4',
'MUSA available': False,
'OpenCV': '4.9.0',
'PyTorch': '2.3.0',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201703\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2023.1-Product Build 20230303 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v3.3.6 (Git Hash '
'86e6af5974177e513fd3fee58425e1063e7f1361)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX512\n'
' - CUDA Runtime 12.1\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
' - CuDNN 8.9.2\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=12.1, '
'CUDNN_VERSION=8.9.2, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
'-fabi-version=11 -fvisibility-inlines-hidden '
'-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
'-DLIBKINETO_NOROCTRACER -DUSE_FBGEMM '
'-DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK '
'-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
'-O2 -fPIC -Wall -Wextra -Werror=return-type '
'-Werror=non-virtual-dtor -Werror=bool-operation '
'-Wnarrowing -Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wno-unused-parameter '
'-Wno-unused-function -Wno-unused-result '
'-Wno-strict-overflow -Wno-strict-aliasing '
'-Wno-stringop-overflow -Wsuggest-override '
'-Wno-psabi -Wno-error=pedantic '
'-Wno-error=old-style-cast -Wno-missing-braces '
'-fdiagnostics-color=always -faligned-new '
'-Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, TORCH_VERSION=2.3.0, '
'USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, '
'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, '
'USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, '
'USE_ROCM_KERNEL_ASSERT=OFF, \n',
'Python': '3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0]',
'TorchVision': '0.18.0',
'numpy_random_seed': 2147483648,
'opencompass': '0.2.4+e3c0448',
'sys.platform': 'linux'}

Reproduces the problem - code/configuration sample

运行quick start中的示例
python run.py --models hf_opt_125m hf_opt_350m --datasets siqa_gen winograd_ppl

Reproduces the problem - command or script

见上一条

Reproduces the problem - error message

运行时的输出:

(opencompass) aidt@aidt-System-Product-Name:~/PycharmProjects/opencompass$ python run.py --models hf_opt_125m hf_opt_350m --datasets siqa_gen winograd_ppl
05/16 15:14:31 - OpenCompass - INFO - Loading siqa_gen: configs/datasets/siqa/siqa_gen.py
05/16 15:14:31 - OpenCompass - INFO - Loading winograd_ppl: configs/datasets/winograd/winograd_ppl.py
05/16 15:14:31 - OpenCompass - INFO - Loading hf_opt_125m: configs/models/opt/hf_opt_125m.py
05/16 15:14:31 - OpenCompass - INFO - Loading hf_opt_350m: configs/models/opt/hf_opt_350m.py
05/16 15:14:31 - OpenCompass - INFO - Loading example: configs/summarizers/example.py
05/16 15:14:31 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
05/16 15:14:31 - OpenCompass - INFO - Partitioned into 2 tasks.
launch OpenICLInfer[opt-125m-hf/siqa,opt-125m-hf/winograd] on GPU 0
0%| | 0/2 [00:00<?, ?it/s]05/16 15:14:32 - OpenCompass - ERROR - /home/aidt/PycharmProjects/opencompass/opencompass/runners/local.py - _launch - 208 - task OpenICLInfer[opt-125m-hf/siqa,opt-125m-hf/winograd] fail, see
outputs/default/20240516_151431/logs/infer/opt-125m-hf/siqa.out
launch OpenICLInfer[opt-350m-hf/siqa,opt-350m-hf/winograd] on GPU 0
50%|██████████████████████▌ | 1/2 [00:01<00:01, 1.24s/it]05/16 15:14:33 - OpenCompass - ERROR - /home/aidt/PycharmProjects/opencompass/opencompass/runners/local.py - _launch - 208 - task OpenICLInfer[opt-350m-hf/siqa,opt-350m-hf/winograd] fail, see
outputs/default/20240516_151431/logs/infer/opt-350m-hf/siqa.out
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
05/16 15:14:33 - OpenCompass - ERROR - /home/aidt/PycharmProjects/opencompass/opencompass/runners/base.py - summarize - 64 - OpenICLInfer[opt-125m-hf/siqa,opt-125m-hf/winograd] failed with code 1
05/16 15:14:33 - OpenCompass - ERROR - /home/aidt/PycharmProjects/opencompass/opencompass/runners/base.py - summarize - 64 - OpenICLInfer[opt-350m-hf/siqa,opt-350m-hf/winograd] failed with code 1
05/16 15:14:33 - OpenCompass - INFO - Partitioned into 4 tasks.
launch OpenICLEval[opt-125m-hf/siqa] on CPU
launch OpenICLEval[opt-125m-hf/winograd] on CPU
launch OpenICLEval[opt-350m-hf/siqa] on CPU
launch OpenICLEval[opt-350m-hf/winograd] on CPU
100%|█████████████████████████████████████████████| 4/4 [00:25<00:00, 6.36s/it]
dataset version metric mode opt-125m-hf opt-350m-hf


siqa - - - - -
winograd - - - - -
05/16 15:14:59 - OpenCompass - INFO - write summary to /home/aidt/PycharmProjects/opencompass/outputs/default/20240516_151431/summary/summary_20240516_151431.txt
05/16 15:14:59 - OpenCompass - INFO - write csv to /home/aidt/PycharmProjects/opencompass/outputs/default/20240516_151431/summary/summary_20240516_151431.csv
检查siqa.out,内容是;
Error: mkl-service + Intel(R) MKL: MKL_THREADING_LAYER=INTEL is incompatible with libgomp.so.1 library.
Try to import numpy first or set the threading layer accordingly. Set MKL_SERVICE_FORCE_INTEL to force it.

尝试设置环境变量:
export MKL_SERVICE_FORCE_INTEL=1
重新运行:
python run.py --models hf_opt_125m hf_opt_350m --datasets siqa_gen winograd_ppl
就一直卡在:
(opencompass) aidt@aidt-System-Product-Name:~/PycharmProjects/opencompass$ python run.py --models hf_opt_125m hf_opt_350m --datasets siqa_gen winograd_ppl
05/16 15:22:14 - OpenCompass - INFO - Loading siqa_gen: configs/datasets/siqa/siqa_gen.py
05/16 15:22:14 - OpenCompass - INFO - Loading winograd_ppl: configs/datasets/winograd/winograd_ppl.py
05/16 15:22:14 - OpenCompass - INFO - Loading hf_opt_125m: configs/models/opt/hf_opt_125m.py
05/16 15:22:14 - OpenCompass - INFO - Loading hf_opt_350m: configs/models/opt/hf_opt_350m.py
05/16 15:22:14 - OpenCompass - INFO - Loading example: configs/summarizers/example.py
05/16 15:22:14 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
05/16 15:22:14 - OpenCompass - INFO - Partitioned into 2 tasks.
launch OpenICLInfer[opt-125m-hf/siqa,opt-125m-hf/winograd] on GPU 0
0%| | 0/2 [00:00<?, ?it/s]

Other information

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.