open-compass / open-compass/opencompass

[Bug] 🐛 Type Err in Humaneval

Open
#923 0 comments 0 reactions 2 assignees View on GitHub

@Leymore is already working on this.

Since Mar 13, 2024.

Dominant language
Python
Stars
7.5k
Forks
869
Avg merge
17h 52m
Merged PRs (30d)
13

Description

Prerequisite
Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment
{
   'CUDA available': True,
   'CUDA_HOME': '/usr/local/cuda-11.7',
   'GCC': 'gcc (GCC) 8.4.1 20200928 (Anolis 8.4.1-1.0.1)',
   'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A100-PCIE-40GB',
   'MMEngine': '0.10.3',
   'MUSA available': False,
   'NVCC': 'Cuda compilation tools, release 11.7, V11.7.99',
   'OpenCV': '4.9.0',
   'PyTorch': '1.13.1+cu117',
   'PyTorch compiling details': 'PyTorch built with:\n'
                                '  - GCC 9.3\n'
                                '  - C++ Version: 201402\n'
                                '  - Intel(R) Math Kernel Library Version '
                                '2020.0.0 Product Build 20191122 for Intel(R) 64 '
                                'architecture applications\n'
                                '  - Intel(R) MKL-DNN v2.6.0 (Git Hash '
                                '52b5f107dd9cf10910aaa19cb47f3abf9b349815)\n'
                                '  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
                                '  - LAPACK is enabled (usually provided by '
                                'MKL)\n'
                                '  - NNPACK is enabled\n'
                                '  - CPU capability usage: AVX2\n'
                                '  - CUDA Runtime 11.7\n'
                                '  - NVCC architecture flags: '
                                '-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86\n'
                                '  - CuDNN 8.5\n'
                                '  - Magma 2.6.1\n'
                                '  - Build settings: BLAS_INFO=mkl, '
                                'BUILD_TYPE=Release, CUDA_VERSION=11.7, '
                                'CUDNN_VERSION=8.5.0, '
                                'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
                                'CXX_FLAGS= -fabi-version=11 -Wno-deprecated '
                                '-fvisibility-inlines-hidden -DUSE_PTHREADPOOL '
                                '-fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM '
                                '-DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK '
                                '-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
                                '-DEDGE_PROFILER_USE_KINETO -O2 -fPIC '
                                '-Wno-narrowing -Wall -Wextra '
                                '-Werror=return-type -Werror=non-virtual-dtor '
                                '-Wno-missing-field-initializers '
                                '-Wno-type-limits -Wno-array-bounds '
                                '-Wno-unknown-pragmas -Wunused-local-typedefs '
                                '-Wno-unused-parameter -Wno-unused-function '
                                '-Wno-unused-result -Wno-strict-overflow '
                                '-Wno-strict-aliasing '
                                '-Wno-error=deprecated-declarations '
                                '-Wno-stringop-overflow -Wno-psabi '
                                '-Wno-error=pedantic -Wno-error=redundant-decls '
                                '-Wno-error=old-style-cast '
                                '-fdiagnostics-color=always -faligned-new '
                                '-Wno-unused-but-set-variable '
                                '-Wno-maybe-uninitialized -fno-math-errno '
                                '-fno-trapping-math -Werror=format '
                                '-Werror=cast-function-type '
                                '-Wno-stringop-overflow, LAPACK_INFO=mkl, '
                                'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
                                'PERF_WITH_AVX512=1, TORCH_VERSION=1.13.1, '
                                'USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, '
                                'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, '
                                'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, '
                                'USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n',
   'Python': '3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]',
   'TorchVision': '0.14.1+cu117',
   'numpy_random_seed': 2147483648,
   'opencompass': '0.2.2+ae0858c',
   'sys.platform': 'linux'
}
Reproduces the problem - code/configuration sample
with read_base():
    from .datasets.humaneval.humaneval_gen import humaneval_datasets

datasets = [
    *humaneval_datasets,
]
Reproduces the problem - command or script
python run.py /path/to/config/config.py -m eval -r /path/to/timestamp/data
Reproduces the problem - error message
AttributeError: 'bool' object has no attribute 'lstrip'
Other information

Due to unexpected output format from LLM, This function humaneval_postprocess_v2 may raise some type errors while handling text = text.lstrip('\n').

def humaneval_postprocess_v2(text: str) -> str:
    """This is an advanced version of previous postprocess to handle more
    situations, better to use this one."""
    try:
        # for chatGLM raw text
        text = eval(text)
    except Exception:
        pass
    text = text.lstrip('\n')

e.g., While the output from LLM is only a 'False', the processing function will directly take it as a bool.
Some type checks or convertions may be required

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.