[Bug] The program frequently stops running during execution

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• The bug has not been fixed in the latest version.

Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

The provided environment information doesn't represent the actual running machines; however, their images and Python environments are consistent, with the only difference being the number of GPUs.

{'CUDA available': True,
'CUDA_HOME': '/usr/local/cuda',
'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0',
'GPU 0,1': 'NVIDIA A100-SXM4-80GB',
'MMEngine': '0.9.1',
'NVCC': 'Cuda compilation tools, release 11.8, V11.8.89',
'OpenCV': '4.8.1',
'PyTorch': '2.1.0',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201703\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2023.1-Product Build 20230303 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v3.1.1 (Git Hash '
'64f6bcbcbab628e96f33a62c3e975f8535a7bde4)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX512\n'
' - CUDA Runtime 11.8\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_37,code=compute_37\n'
' - CuDNN 8.7\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=11.8, '
'CUDNN_VERSION=8.7.0, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
'-fabi-version=11 -fvisibility-inlines-hidden '
'-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
'-DLIBKINETO_NOROCTRACER -DUSE_FBGEMM '
'-DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK '
'-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
'-O2 -fPIC -Wall -Wextra -Werror=return-type '
'-Werror=non-virtual-dtor -Werror=bool-operation '
'-Wnarrowing -Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wno-unused-parameter '
'-Wno-unused-function -Wno-unused-result '
'-Wno-strict-overflow -Wno-strict-aliasing '
'-Wno-stringop-overflow -Wno-psabi '
'-Wno-error=pedantic -Wno-error=old-style-cast '
'-Wno-invalid-partial-specialization '
'-Wno-unused-private-field '
'-Wno-aligned-allocation-unavailable '
'-Wno-missing-braces -fdiagnostics-color=always '
'-faligned-new -Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Werror=cast-function-type '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, '
'TORCH_DISABLE_GPU_ASSERTS=ON, '
'TORCH_VERSION=2.1.0, USE_CUDA=ON, USE_CUDNN=ON, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, '
'USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, '
'USE_OPENMP=ON, USE_ROCM=OFF, \n',
'Python': '3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]',
'TorchVision': '0.16.0',
'numpy_random_seed': 2147483648,
'opencompass': '0.1.8+19ad7f9',
'sys.platform': 'linux'}

Reproduces the problem - code/configuration sample

It occurs in the majority of the runtime, but due to the absence of logs, I'm not sure what happened. All I know is the observed behavior: in the presence of unfinished tasks, the GPU no longer seems to be utilized. It appears that there may be an issue with task scheduling.

Reproduces the problem - command or script

python run.py --models llama2_13b_hf --summarizer leaderboard -r -l  -w ./outputs/main --datasets \
SuperGLUE_COPA_ppl piqa_ppl siqa_ppl strategyqa_gen bbh_gen mmlu_ppl SuperGLUE_MultiRC_ppl race_ppl drop_gen \
obqa_ppl squad20_gen Xsum_gen  lambada_gen leval longbench civilcomments_clp tydiqa_gen XLSum_gen \
SuperGLUE_BoolQ_ppl commonsenseqa_ppl SuperGLUE_AX_b_ppl SuperGLUE_AX_g_ppl storycloze_ppl triviaqa_gen

Reproduces the problem - error message

Only the GPU is not being utilized, and this persists for a considerable duration

Other information

Initially, all 8 GPUs are operational, but as time progresses, they gradually cease to function.

It's important to note that the tasks haven't completed; the tasks (containers) automatically terminate even before their execution is finished.

This issue arises both in the initial run and in subsequent runs (since the GPU is already unoccupied after the first run).

Additionally, it seems that there is a specific sequence for the execution of 'commonsenseqa_ppl'? I encounter failures each time I run it, and it requires running it again for normal operation.

Error message like this:

Traceback (most recent call last):
  File "-/opencompass/opencompass/tasks/openicl_infer.py", line 148, in <module>
    inferencer.run()
  File "-/opencompass/opencompass/tasks/openicl_infer.py", line 78, in run
    self._inference()
  File "-opencompass/opencompass/tasks/openicl_infer.py", line 96, in _inference
    retriever = ICL_RETRIEVERS.build(retriever_cfg)
  File "-/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
    return self.build_func(cfg, *args, **kwargs, registry=self)
  File "-/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
    obj = obj_cls(**args)  # type: ignore
  File "-/opencompass/opencompass/openicl/icl_retriever/icl_mdl_retriever.py", line 73, in __init__
    super().__init__(dataset, ice_separator, ice_eos_token, ice_num,
  File "/-/opencompass/opencompass/openicl/icl_retriever/icl_topk_retriever.py", line 78, in __init__
    self.model = SentenceTransformer(sentence_transformers_model_name)
  File "-lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py", line 87, in __init__
    snapshot_download(model_name_or_path,
  File "-/python3.10/site-packages/sentence_transformers/util.py", line 491, in snapshot_download
    path = cached_download(**cached_download_args)
  File "-/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
  File "-/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 757, in cached_download
    raise LocalEntryNotFoundError(
huggingface_hub.utils._errors.LocalEntryNotFoundError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.
[2023-11-15 16:32:58,769] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 68088) of binary: -/bin/python
Traceback (most recent call last):
  File "-bin/torchrun", line 33, in <module>
    sys.exit(load_entry_point('torch==2.1.0', 'console_scripts', 'torchrun')())
  File "-/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "-/python3.10/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "-b/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "-/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "-/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
-opencompass/opencompass/tasks/openicl_infer.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-11-15_16:32:58
  host      : t-20231115174409-7gr47-worker-0.t-20231115174409-7gr47-worker.mlplatform-customtask.svc.cluster.local
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 68088)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

However, running the script once all tasks have completed seems to allow for normal execution.

In summary, I have found that failures in certain datasets (tasks) can be resolved by rerunning without any code modifications. There seems to be some issues with this.