deepspeedai / deepspeedai/DeepSpeed

[BUG] Inference failed serveral times

Open
#3,181 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug inference
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Describe the bug
When I run my inference code with deepspeed.init_inference(). It only works a few times with num_gpus=2 (num_gpus>2 always fails, num_gpus=2 sometimes fails). Following this link https://www.deepspeed.ai/tutorials/inference-tutorial/

To Reproduce
Steps to reproduce the behavior:
0. Installation

conda create -n py38 -y python=3.8
conda activate py38 
conda install -c "nvidia/label/cuda-11.7.0" cuda-toolkit   # default installed nvcc in my machine is 10.1
pip install torch==1.13.0+cu117 torchvision==0.14.0+cu117 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/cu117
pip install ninja 

git clone https://github.com/microsoft/DeepSpeed/
cd DeepSpeed
rm -rf build
TORCH_CUDA_ARCH_LIST="8.6" DS_BUILD_TRANSFORMER_INFERENCE=1 pip install . \
--global-option="build_ext" --global-option="-j8" --no-cache -v \
--disable-pip-version-check 2>&1 | tee build.log
  1. My inference script
# run.py 

import deepspeed
import torch 
import os 
from transformers import AutoModel

local_rank = int(os.getenv('LOCAL_RANK', '0'))
world_size = int(os.getenv('WORLD_SIZE', '1'))

model = AutoModel.from_pretrained("vinai/phobert-base").eval()
model.to(local_rank)

# Initialize the DeepSpeed-Inference engine
ds_engine = deepspeed.init_inference(model,
                                 mp_size=world_size,
                                 checkpoint=None,
                                 dtype=torch.float)

model = ds_engine.module

input_ids = torch.LongTensor([[0, 1, 2, 3]]).to(local_rank)

with torch.no_grad():
    output = model(input_ids=input_ids)

print(output.last_hidden_state)
  1. How to run the script
deepspeed --num_gpus=2 run.py

Expected behavior
Out when running successfully

(py38) root@quangthd-7c6fb44f48-qfdg7:/home/workspace/train_exp# deepspeed --num_gpus=2 run_1.py
[2023-04-11 02:37:20,770] [WARNING] [runner.py:181:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2023-04-11 02:37:20,864] [INFO] [runner.py:527:main] cmd = /root/miniconda3/envs/py38/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMV19 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None run_1.py
[2023-04-11 02:37:23,948] [INFO] [launch.py:126:main] 0 NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.13.4-1+cuda11.7
[2023-04-11 02:37:23,948] [INFO] [launch.py:126:main] 0 NCCL_VERSION=2.13.4-1
[2023-04-11 02:37:23,948] [INFO] [launch.py:126:main] 0 NV_LIBNCCL_PACKAGE_VERSION=2.13.4-1
[2023-04-11 02:37:23,949] [INFO] [launch.py:126:main] 0 NV_LIBNCCL_PACKAGE=libnccl2=2.13.4-1+cuda11.7
[2023-04-11 02:37:23,949] [INFO] [launch.py:126:main] 0 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
[2023-04-11 02:37:23,949] [INFO] [launch.py:126:main] 0 NV_LIBNCCL_PACKAGE_NAME=libnccl2
[2023-04-11 02:37:23,949] [INFO] [launch.py:126:main] 0 NV_LIBNCCL_DEV_PACKAGE_VERSION=2.13.4-1
[2023-04-11 02:37:23,949] [INFO] [launch.py:133:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2023-04-11 02:37:23,949] [INFO] [launch.py:139:main] nnodes=1, num_local_procs=2, node_rank=0
[2023-04-11 02:37:23,949] [INFO] [launch.py:150:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
[2023-04-11 02:37:23,949] [INFO] [launch.py:151:main] dist_world_size=2
[2023-04-11 02:37:23,949] [INFO] [launch.py:153:main] Setting CUDA_VISIBLE_DEVICES=0,1
[2023-04-11 02:37:35,509] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.8.3+4d27225f, git-hash=4d27225f, git-branch=master
[2023-04-11 02:37:35,515] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-04-11 02:37:35,516] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
[2023-04-11 02:37:35,520] [INFO] [comm.py:586:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2023-04-11 02:37:35,728] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.8.3+4d27225f, git-hash=4d27225f, git-branch=master
[2023-04-11 02:37:35,730] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-04-11 02:37:35,731] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
AutoTP:  [(<class 'transformers.models.roberta.modeling_roberta.RobertaLayer'>, ['output.dense'])]
AutoTP:  [(<class 'transformers.models.roberta.modeling_roberta.RobertaLayer'>, ['output.dense'])]
tensor([[[-0.1506,  0.1654, -0.2740,  ..., -0.0372,  0.0528, -0.4088],
         [-0.0466,  0.0702,  0.1521,  ..., -0.3913,  0.2049, -0.2464],
         [-0.2765, -0.1203, -0.7273,  ..., -0.2264, -0.0797, -0.4175],
         [-0.2209, -0.2705, -0.7297,  ..., -0.3494,  0.2433, -0.7732]]],
       device='cuda:1')
tensor([[[-0.1506,  0.1654, -0.2740,  ..., -0.0372,  0.0528, -0.4088],
         [-0.0466,  0.0702,  0.1521,  ..., -0.3913,  0.2049, -0.2464],
         [-0.2765, -0.1203, -0.7273,  ..., -0.2264, -0.0797, -0.4175],
         [-0.2209, -0.2705, -0.7297,  ..., -0.3494,  0.2433, -0.7732]]],
       device='cuda:0')
[2023-04-11 02:37:39,011] [INFO] [launch.py:329:main] Process 12518 exits successfully.
[2023-04-11 02:37:39,012] [INFO] [launch.py:329:main] Process 12517 exits successfully.

ds_report output
To create bug, you can run the script several times or increase the --num_gpus
Here is the log

(py38) root@quangthd-7c6fb44f48-qfdg7:/home/workspace/train_exp# deepspeed --num_gpus=3 run_1.py
[2023-04-11 02:38:38,952] [WARNING] [runner.py:181:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2023-04-11 02:38:39,050] [INFO] [runner.py:527:main] cmd = /root/miniconda3/envs/py38/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMl19 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None run_1.py
[2023-04-11 02:38:41,915] [INFO] [launch.py:126:main] 0 NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.13.4-1+cuda11.7
[2023-04-11 02:38:41,915] [INFO] [launch.py:126:main] 0 NCCL_VERSION=2.13.4-1
[2023-04-11 02:38:41,915] [INFO] [launch.py:126:main] 0 NV_LIBNCCL_PACKAGE_VERSION=2.13.4-1
[2023-04-11 02:38:41,915] [INFO] [launch.py:126:main] 0 NV_LIBNCCL_PACKAGE=libnccl2=2.13.4-1+cuda11.7
[2023-04-11 02:38:41,915] [INFO] [launch.py:126:main] 0 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
[2023-04-11 02:38:41,915] [INFO] [launch.py:126:main] 0 NV_LIBNCCL_PACKAGE_NAME=libnccl2
[2023-04-11 02:38:41,915] [INFO] [launch.py:126:main] 0 NV_LIBNCCL_DEV_PACKAGE_VERSION=2.13.4-1
[2023-04-11 02:38:41,916] [INFO] [launch.py:133:main] WORLD INFO DICT: {'localhost': [0, 1, 2]}
[2023-04-11 02:38:41,916] [INFO] [launch.py:139:main] nnodes=1, num_local_procs=3, node_rank=0
[2023-04-11 02:38:41,916] [INFO] [launch.py:150:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2]})
[2023-04-11 02:38:41,916] [INFO] [launch.py:151:main] dist_world_size=3
[2023-04-11 02:38:41,916] [INFO] [launch.py:153:main] Setting CUDA_VISIBLE_DEVICES=0,1,2
[2023-04-11 02:39:07,402] [INFO] [launch.py:297:sigkill_handler] Killing subprocess 13678
[2023-04-11 02:39:07,603] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.8.3+4d27225f, git-hash=4d27225f, git-branch=master
[2023-04-11 02:39:07,605] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-04-11 02:39:07,606] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
[2023-04-11 02:39:07,622] [INFO] [launch.py:297:sigkill_handler] Killing subprocess 13679
[2023-04-11 02:39:07,840] [INFO] [launch.py:297:sigkill_handler] Killing subprocess 13680
[2023-04-11 02:39:07,840] [ERROR] [launch.py:303:sigkill_handler] ['/root/miniconda3/envs/py38/bin/python', '-u', 'run_1.py', '--local_rank=2'] exits with return code = -9

ds_report

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  please install triton==1.0.0 if you want to use sparse attention
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [YES] ...... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/root/miniconda3/envs/py38/lib/python3.8/site-packages/torch']
torch version .................... 1.13.1+cu117
deepspeed install path ........... ['/root/miniconda3/envs/py38/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.8.3+4d27225f, 4d27225f, master
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.7
deepspeed wheel compiled w. ...... torch 1.13, cuda 11.7

System info (please complete the following information):

  • OS: Ubuntu 18.04.6 LTS
  • GPU: 1 machine 12 GPUs A16
  • Deepspeed version: 0.8.3
  • Transformers version: 4.27.4
  • Python version: 3.8.16

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided run.py reproduction and the DeepSpeed inference tutorial, then run the listed deepspeed --num_gpus commands and inspect the ds_report output. Done means determining why repeated or three-GPU inference exits with -9 and making the documented setup run reliably across the tested GPU counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.