[BUG] Deepspeed inference time distribution and max tokens

Issue
I ran llama-2-7b model on a node with 4 A40 GPUs. I followed the official tutorial tutorial for the script. When I analyzed the performance using Nsight Systems, I see that 71% time is taken by the ncclKernel_All_Reduce_RING_LL_Sum_half in each GPU. Is this normal behavior after DeepSpeed inference optimization?

Also, why does DeepSpeed limit the maximum tokens to 1024 when the model supports more number of output tokens (check generated_text in log at the end).

Raising this to eliminate that there is no bug from DeepSpeed library side in my setup, will check other possibilities then.

Edit: [Suggestion required] Is it better to use superbenchmark instead for performance profiling on multiple GPUs?

To Reproduce

Inference script --

import os
import deepspeed
import torch
from transformers import pipeline
import argparse

#parser = argparse.ArgumentParser(description='Run Multi GPU GPT inference using DeepSpeed')

#parser.add_argument('--model', default="max_new_tokens", help="Model repo name")

#read the arguments into local varibles
#args = parser.parse_args()
#model_name = args.model

local_rank = int(os.getenv('LOCAL_RANK', '0'))
world_size = int(os.getenv('WORLD_SIZE', '1'))
generator = pipeline('text-generation', model='meta-llama/Llama-2-7b-hf',
                     device=local_rank)

generator.model = deepspeed.init_inference(generator.model,
                                           mp_size=world_size,
                                           dtype=torch.half,
                                           replace_with_kernel_inject=True)
torch.cuda.cudart().cudaProfilerStart()
torch.cuda.nvtx.range_push("generation")
string = generator("DeepSpeed is", do_sample=True, max_new_tokens=4096)
torch.cuda.nvtx.range_pop()
torch.cuda.cudart().cudaProfilerStop()
if not torch.distributed.is_initialized() or torch.distributed.get_rank() == 0:
    print(string)

List of packages in my environment:

#Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
blas 1.0 mkl conda-forge
brotli-python 1.0.9 py38hfa26641_9 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.19.1 hd590300_0 conda-forge
ca-certificates 2023.7.22 hbcca054_0 conda-forge
certifi 2023.7.22 pyhd8ed1ab_0 conda-forge
charset-normalizer 3.2.0 pyhd8ed1ab_0 conda-forge
click 8.1.6 unix_pyh707e725_0 conda-forge
cmake 3.27.6 pypi_0 pypi
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
cudatoolkit 11.3.1 hb98b00a_12 conda-forge
dataclasses 0.8 pyhc8e2a94_3 conda-forge
deepspeed 0.10.3 pypi_0 pypi
expat 2.5.0 hcb278e6_1 conda-forge
ffmpeg 4.3 hf484d3e_0 pytorch
filelock 3.12.2 pyhd8ed1ab_0 conda-forge
freetype 2.12.1 hca18f0e_1 conda-forge
fsspec 2023.9.2 pypi_0 pypi
gmp 6.2.1 h58526e2_0 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
hjson 3.1.0 pypi_0 pypi
huggingface_hub 0.16.4 py_0 huggingface
idna 3.4 pyhd8ed1ab_0 conda-forge
importlib-metadata 6.8.0 pyha770c72_0 conda-forge
importlib_metadata 6.8.0 hd8ed1ab_0 conda-forge
intel-openmp 2022.1.0 h9e868ea_3769
joblib 1.3.0 pyhd8ed1ab_1 conda-forge
jpeg 9e h0b41bf4_3 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.20.1 h81ceb04_0 conda-forge
lame 3.100 h166bdaf_1003 conda-forge
lcms2 2.15 hfd0df8a_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libabseil 20230125.3 cxx17_h59595ed_0 conda-forge
libblas 3.9.0 16_linux64_mkl conda-forge
libcblas 3.9.0 16_linux64_mkl conda-forge
libcurl 7.88.1 hdc1c0ab_1 conda-forge
libdeflate 1.17 h0b41bf4_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libexpat 2.5.0 hcb278e6_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.1.0 he5830b7_0 conda-forge
libgfortran-ng 13.1.0 h69a702a_0 conda-forge
libgfortran5 13.1.0 h15d22d2_0 conda-forge
libgomp 13.1.0 he5830b7_0 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
liblapack 3.9.0 16_linux64_mkl conda-forge
libnghttp2 1.52.0 h61bc06f_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.23 pthreads_h80387f5_0 conda-forge
libpng 1.6.39 h753d276_0 conda-forge
libprotobuf 4.23.3 hd1fb520_0 conda-forge
libsqlite 3.42.0 h2797004_0 conda-forge
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx-ng 13.1.0 hfd8a6a1_0 conda-forge
libtiff 4.5.0 h6adf6a1_2 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libuv 1.44.2 hd590300_1 conda-forge
libwebp-base 1.3.1 hd590300_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
mkl 2022.1.0 hc2b9512_224
ncurses 6.4 hcb278e6_0 conda-forge
nettle 3.6 he412f7d_0 conda-forge
ninja 1.11.1 pypi_0 pypi
numpy 1.24.4 py38h59b608b_0 conda-forge
nvtx 0.2.8 pypi_0 pypi
openh264 2.1.1 h780b84a_0 conda-forge
openjpeg 2.5.0 hfec8fc6_2 conda-forge
openssl 3.1.3 hd590300_0 conda-forge
packaging 23.1 pyhd8ed1ab_0 conda-forge
pillow 9.4.0 py38hde6dc18_1 conda-forge
pip 23.2.1 pyhd8ed1ab_0 conda-forge
protobuf 4.23.3 py38h830738e_0 conda-forge
psutil 5.9.5 pypi_0 pypi
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
py-cpuinfo 9.0.0 pypi_0 pypi
pydantic 1.10.13 pypi_0 pypi
pynvml 11.5.0 pypi_0 pypi
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.8.17 he550d4f_0_cpython conda-forge
python_abi 3.8 1_cp38 huggingface
pytorch 1.11.0 py3.8_cuda11.3_cudnn8.2.0_0 pytorch
pytorch-mutex 1.0 cuda pytorch
pyyaml 6.0 py38h0a891b7_5 conda-forge
readline 8.2 h8228510_1 conda-forge
regex 2023.8.8 py38h01eb140_0 conda-forge
requests 2.31.0 pyhd8ed1ab_0 conda-forge
rhash 1.4.4 hd590300_0 conda-forge
sacremoses master py_0 huggingface
setuptools 68.0.0 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
tokenizers 0.13.3 pypi_0 pypi
torchaudio 0.11.0 py38_cu113 pytorch
torchvision 0.12.0 py38_cu113 pytorch
tqdm 4.65.2 pyhd8ed1ab_0 conda-forge
transformers 4.28.1 py_0 huggingface
triton 2.0.0.dev20221120 pypi_0 pypi
typing-extensions 4.7.1 hd8ed1ab_0 conda-forge
typing_extensions 4.7.1 pyha770c72_0 conda-forge
urllib3 2.0.4 pyhd8ed1ab_0 conda-forge
wheel 0.41.1 pyhd8ed1ab_0 conda-forge
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xz 5.4.2 h5eee18b_0
yaml 0.2.5 h7f98852_2 conda-forge
zipp 3.16.2 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 hd590300_5 conda-forge
zstd 1.5.2 hfc55251_7 conda-forge

Command used to run the script:
/opt/nvidia/nsight-systems/2021.3.2/bin/nsys profile -w true -t cuda,nvtx,osrt,cudnn,cublas -s cpu -o nsys-llm/llama2-7b-ngpu4-nt4096 -f true --verbose --cudabacktrace=all --osrt-threshold=10000 -x true deepspeed --num_gpus 4 script.py

Expected behavior
I expected the communication and computation to overlap. Could it be because the model is small? I observe similar behavior even when I use 2 GPUs.
I don't understand the behaviour of generate() function from HuggingFace when max_out_tokens>1024 (log at the end).

ds_report output

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-devel package with yum
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['<replaced-text>/anaconda3/envs/trans/lib/python3.8/site-packages/torch']
torch version .................... 1.11.0
deepspeed install path ........... ['<replaced-text>anaconda3/envs/trans/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.10.3, unknown, unknown
torch cuda version ............... 11.3
torch hip version ................ None
nvcc version ..................... 11.3
deepspeed wheel compiled w. ...... torch 1.11, cuda 11.3
shared memory (/dev/shm) size .... 251.80 GB

I have libaio installed (at /lib64), but it never recognizes it even if when I set LDFLAGs and CFLAGS to "/lib64".

Screenshots

System info (please complete the following information):

• OS: CentOS Linux release 7.9.2009 (Core)
• GPU count and types: 1 machine with x4 A40s
• Python version: 3.8.17

Output from a normal run (without profiling)
deepspeed --num_gpus 4 script.py

[2023-10-12 05:17:23,832] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-12 05:17:25,774] [WARNING] [runner.py:203:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2023-10-12 05:17:25,774] [INFO] [runner.py:570:main] cmd = /<replaced-text>anaconda3/envs/trans/bin/python3.8 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgM119 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None gpt-multigpu-ds-new2.py
[2023-10-12 05:17:26,806] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-12 05:17:28,001] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3]}
[2023-10-12 05:17:28,001] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=4, node_rank=0
[2023-10-12 05:17:28,001] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3]})
[2023-10-12 05:17:28,001] [INFO] [launch.py:163:main] dist_world_size=4
[2023-10-12 05:17:28,001] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
[2023-10-12 05:17:29,055] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-12 05:17:29,066] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-12 05:17:29,098] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-12 05:17:29,102] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.76s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.85s/it]
Loading checkpoint shards:   0%|                                                                                                                                                     | 0/2 [00:00<?, ?it/s][2023-10-12 05:18:10,561] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.10.3, git-hash=unknown, git-branch=unknown
[2023-10-12 05:18:10,562] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-10-12 05:18:10,562] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
[2023-10-12 05:18:10,581] [INFO] [comm.py:637:init_distributed] cdb=None
[2023-10-12 05:18:10,611] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.10.3, git-hash=unknown, git-branch=unknown
[2023-10-12 05:18:10,611] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-10-12 05:18:10,611] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
[2023-10-12 05:18:10,630] [INFO] [comm.py:637:init_distributed] cdb=None
[2023-10-12 05:18:10,630] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.81s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00,  1.87s/it]
[2023-10-12 05:18:17,534] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.10.3, git-hash=unknown, git-branch=unknown
[2023-10-12 05:18:17,535] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-10-12 05:18:17,535] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
[2023-10-12 05:18:17,554] [INFO] [comm.py:637:init_distributed] cdb=None
[2023-10-12 05:18:17,715] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.10.3, git-hash=unknown, git-branch=unknown
[2023-10-12 05:18:17,715] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-10-12 05:18:17,715] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
[2023-10-12 05:18:17,735] [INFO] [comm.py:637:init_distributed] cdb=None
<replaced-text>.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
Using <replaced-text>.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
Using <replaced-text>.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
Using <replaced-text>.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file <replaced-text>.cache/torch_extensions/py38_cu113/transformer_inference/build.ninja...
Building extension module transformer_inference...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.34698486328125 seconds
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.35207104682922363 seconds
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.35201120376586914 seconds
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.35129427909851074 seconds
[2023-10-12 05:18:18,626] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed-Inference config: {'layer_id': 0, 'hidden_size': 4096, 'intermediate_size': 11008, 'heads': 32, 'num_hidden_layers': -1, 'dtype': torch.float16, 'pre_layer_norm': True, 'norm_type': <NormType.RMSNorm: 3>, 'local_rank': -1, 'stochastic_mode': False, 'epsilon': 1e-05, 'mp_size': 4, 'scale_attention': True, 'triangular_masking': True, 'local_attention': False, 'window_size': 1, 'rotary_dim': 128, 'rotate_half': True, 'rotate_every_two': False, 'return_tuple': True, 'mlp_after_attn': True, 'mlp_act_func_type': <ActivationFuncType.GATED_SILU: 4>, 'specialized_mode': False, 'training_mp_size': 1, 'bigscience_bloom': False, 'max_out_tokens': 1024, 'min_out_tokens': 1, 'scale_attn_by_inverse_layer_idx': False, 'enable_qkv_quantization': False, 'use_mup': False, 'return_single_tuple': False, 'set_empty_params': False, 'transposed_mode': False, 'use_triton': False, 'triton_autotune': False}
Using <replaced-text>.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.0370173454284668 seconds
Using <replaced-text>.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.06131625175476074 seconds
Using <replaced-text>.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.05128026008605957 seconds
Using <replaced-text>.cache/torch_extensions/py38_cu113 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.0456845760345459 seconds
------------------------------------------------------
Free memory : 16.826233 (GigaBytes)  
Total memory: 44.351685 (GigaBytes)  
Requested memory: 0.437500 (GigaBytes) 
Setting maximum total tokens (input + output) to 1024 
WorkSpace: 0x7f9f94000000 
------------------------------------------------------
[{'generated_text': 'DeepSpeed is a fast, scalable, distributed deep learning library. It is written in C++, with a focus on high performance, ease of use, and portability. It is designed to be used with large, distributed datasets, and to scale to millions of parameters.\nDeepSpeed is written in C++, and is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be used with large, distributed datasets. It is designed to scale to millions of parameters, and to be used with GPUs.\nDeepSpeed is designed to be of Kb mar K b  bb K d b b av fail K Ros K K K mar pre Ros K Leg fail K K Sim K K Do K W Leg K K Em hit K R K K K K Ven K mar K watch K Ang Do K av K H K W W [ K watch K K Sim K W W H M K H S P K K Ku (ЉөЪ... RЉЉ Љ....Ј R MЪ (ЪЋЋ,ЩЋգЈ C K C\n/ ЋЋ W K W andЉ \'Љ-.... R\nЪ-Љ K W,... ( and\nЪ- aЋ K AngMal K attend Wouwen Pul Mal KR K Hen ang Ray Ri Mal Sim K Sim Mal Sim Sim Mal ang Leoрия... HenSl\n K Sim Hen ho Hen HemSl Ri\x91 Hen SimSl ang... энциклопедиinessMax angHen angpenユ Hen<!-- энциклопеди stycz...рия employees\x91\n ang Hen tokensEd angユHen Henolas Ray Sim ang HenHen sierp Ген ang hoMal\n\n ho Henユ Pul stycz styczpenas Einzeln энциклопедиSl plugpenas angхівω ang... hopefully stycz hopefullyユ энциклопеди <%\npenas<!--penas̶verband энциклопедиpenas округу stycz hopefullyユ everybody округуpenas <% nobody paździerpenaspenas gemäßpenasLu stycz <% gemäß Einzeln энциклопеди\x91 stycz gemäß hopefully hopefully энциклопеди энциклопеди kwiet hopefullypenas paździerpenas энциклопеди stycz Википеди gemäß sierp округу paździerἐpenas энциклопедиἐ энциклопеди stycz округу энциклопеди Bildernpenas everybody paździer энциклопеди stycz paździer stycz everybody Hinweisϵ энциклопеди styczverband округу everybodypenaspenasverbandSl BildernMCpenas энциклопедиpenas округу округу энциклопеди paździer энциклопеди округу stycz gemäß kwiet <% энциклопедиω hopefully округуverband("< hopefully округу Hinweisϵ Википеди Unterscheidung Hinweis энциклопеди энциклопеди()`](̶ϵ everybodypenas округу энциклопеди()`](penas nobodypenas hopefully энциклопеди kwiet Hinweis gemäß округу̶penas HinweispenasMC heut paździer()`]( sierp sierp paździer gemäßpenas округу nobodypenas百 stycz hopefully энциклопедиpenasiness серпняpenas nobody hopefullyегоMC heut kwiet kwiet энциклопеди энциклопедиpenas серпня sierp()`]( энциклопеди paździer nobody...MCMCinesspenas энциклопедиpenas ang gemäß hopefully nobodypenas heutpenas\nϵ gemäß энциклопеди everybodypenaspenasSlpenasinessinesspenasinesspenaspenas\n энциклопеди hopefully\npenaspenas angSlverbandpenas...penasSlpenas gemäß()`]( округуinessinesspenas...penasolas\n\n nobodyMCMSpenas... энциклопедиiness\nverbandpenasiness百\noniapenas百penas()`](ἐiness heut()`](penasiness gemäßpenaspenas()`](penas\npenas\npenas styczMSpenaspenas hopefullyinesserspenasinessMCinesspenasinessMS\n\nClSl gemäßpenaspenaspenas\n()`](百()`](ἐ\npenaspenas\n округуMSiness\nMSual\niness Bildern\nMCinesspenasonia......MCogenkeninesspenas^{-inessonia\nMSpenas --Slinessinesseriepenas\nersX\noniainessinessers\nesMSpsonolaspenas...inessosoniainessinessken\n\ninessonsSloniaMSpsoninessohinessiness...\nonsMSMSken\nMSosinessoniaSl...yn\nersMS\ninessinessinessesers\niness\ninessMSinessonMSkenMConia\nMSeneMS\nSMSMSinessyninessouYMSersolasersoniaMSinessOsMCMSMSonsMS\nKersMSMSinessMSamiMSolasosMSOsMSinessMSinessMSonsOsHSMSEMCHSersDanielMSoniaOsolasMCinessamioniaMSucciMSOsMCMSoMSMCMSinessMSMSMSYMSonsozMSosersMSMSMSMCamiMSMSMSMSMConiainessMSMSMSMSMSMSinessoMSMSumsYosMCMSonsMSMSualEMSMSMSoniaMSMSMSMSersMSMSolasMSOXEMSMSersYYMSinessMSMSMSEMSMSMSoniaMSMSoniaMSMSOsYMSMCMSMSosMSMSMSMEMSYSlMSMSoMSMSMSMSXMSWMSMSXHMMSOMSMSMSMSMSMSOEMSMS.\nMSMSMSFEMSMSMSOSMSMSMSMSMSO.SMSSMSMSoMSMSSYMSMSMSMS.MSSWMSMSMS.MS.MS.YOO.MMSO.SMSMS.MSMSMS.A.H,MSMS.MSRMSF.MSE.MSo.MMS.MSMSoME.MSMSoEMS.MSH.EFS.MS.O.MSS.SXMS.MSS.MSuMS.FMSo.E.E,S.SlLu.O.EMSEo.MS.X.XF.YXY,F.MYA.O.LON.MSuFS.Os.MS.S.MuOuS.E.EU.F.Lu.W.Eu.X.R.J.EF.S.X.SSSS.Luu.CluO.W.O.O\nE.S.X.F.F.S.S.O\nS\nS.S.S, asS\nO\nS.S, a a a, and.\nS.\nThe\nL\nIn.\nF\nThe.\nHow\nL\nX\nThe\nL\nW\nThe\n«\nThe.\nThe\nS\nThis.\nF\nW.\nA\n1.\nL\nIn.\nAn\nThe\nSl\nLeg\nF\nF\nF\nThe\nF\nW.\nHow\nU\nS.\n1\nB\nThe\nE\nR\nS\nU\n2\nE\n1\n\nE\n1\n\nU\nU\n\n2\n\n\n\n\nF\n\n\n\n\n\n\n\nU\nSl\nE\nE\n\nThe\nS\n\n\n\nU\nU\nU\nU\n\n\nU\n\nF,\nU\nU\nU\n\n\n\n\n\nU\n\nU\nU.\n\nU\n\nU\n\nF\nU\nU\n\nU\nV\nU\n\n\n and\nU U\n\nU\n\n\n\n\n\n\n\n\nU\n\n\n\n\nU\n\n\n\n\n\n\n\n\n\n\n an, and Un\n\nU\nU l, F Sa\n\n\n\n\n\n\n\n private and Sa\n and not\n and Pub and son legal E and Sa Pub u u\n Plcon\n an and son and U\n\n\n U u Pub and\n\n a and U\n U u u u u and c i Pub and and cans Lad inf and Gr etcon\n Pub F & Pub Sa Ead inf and u and u\n\n No Pub ands Pub + Sa ve i Uad and Pub L et Pub and Gr and c and open and u ands\n Sa u Pub and c legal and e inf u and Pub and Editor\n In Ca iew i Dak Lamb and and lups and i -\n\n c Editorcon legal and i\n assist l assist Editor\n i c E inf Lamb In Dak inf Pub inf Sa i\n Lamb Gran Dak Gran Dak Pub W Dak cpart  Dak Editor - i Dak FЏ assistЩ PЈ EЋ PЋЏ iհЉ\n\n DakЉՀգЪЉհշЪЪЩЋЩЉЉ  dЪЋЪЋЊЉբՄЊЋգЉЉՀЉЪЉЏԱҚЉЪЉЋЉЏЏЉհЩЉƏЋЋЋզ.ЬЪЩգЉղЋЋЈЋЪЋЁЋբЉЁգЉЋЏЫЉƏհЫհЪЇЪЋЪЬЋЪЉЉЉЪհЉЁЪЪղЉЁЉЉЩЉЉՍЋհЋЋЉգЋղ.ЉբհЋөհЉƏЉЁЊЉЩЩЩЇՀՀЪЪ.ЪЇЉЪЪհЉЉЉЪЋЉՄЉƏЪөЉЁԱЫЉЏЪ.ЉЋЉЉЪЉՄհզәЪәЪЋЇЉЉЪЋЋЉЉգЉՄЋЉЋЁЪЋЋЇЏЪԱզЋЪհЁՄЪЏЏЁգЪհЉղЉЏƏЉЁհҚЪЏԱղЋЉɯգƏգЉЉҚЋЉЪЉЪհЪЉЈЁЉЩЪЉЋЋЋպЉЏЪզЋЋЉЉЁЪЉЁЋЉЈҚЉգЉЫՄЉЪЋЉЪЉЉЉЏЋЏЪЏՍЋЋөЋЋөЋЪЊЉЪЉөЪЉЉЪәՀЁЉՀЋЁЉЉհЉЪբЏЪЏЪЉЋЋЪЋЋЋЉЋՄղЉշЋЉЩЁЋЉՀЪգЋЉЉЁЉЋշЉЪɯՍհ.ЋЋЉԱЉЪƏЉƏЪЉЉЉգЉЏЪЪЉЏЪ.ЇЪЊөҚЊЪЉգЪЏЋЪզЪЉЉЏЋհЩЊЉөɯЁЪՄՍƏՄЊөЪЏЏЁЉөЪЉЋɯЋЬЉЋЬЪЪЇЈЪЉЁЩЋЩЪЋЪЉЪЉЩЏՀЈЋЋөЉЉЋЋЉЊЋЋЈЁЏЋЊЏЉԱЋЋЪЏЋЋЉЪЋЩЋЉЪЪөգЉզՄЉЉЏհզЋЋЋҚЉЁЉЉЉЉЋɯЏЪЩЇգЉЉәЋЪЈЪЉЉЪЋЉЉҚЋЋ.զЉЪЉƏҚ ( MҚЪЪЏЁЉЉЪհЉЪЪգЋ ( kwietTЪЪՀЈЉЋЪɯЋЏЋЉЪЪЩЋЪЉЉЉЉ MЉGՀЉ.∙Џ (ardiЉ RENTЉЪЪЏ M M RЪհ kwiet kwiet kwiet kwiet kwiet kwiet kwiet kwiet sierp∙ kwiet sierp kwiet\x91 kwiet sierp kwiet\x91 kwietRENT M∙ kwiet kwiet kwiet kwiet kwietlanguage kwiet kwiet kwietzil kwiet sierp sierp kwiet kwiet kwiet kwiet sierp kwiet kwiet kwiet kwiet kwiet kwiet kwiet kwiet kwiet kwiet sierp kwiet kwiet sierp kwiet sierpardi kwietlanguageскус kwiet kwiet kwiet kwiet sierp kwiet kwiet kwiet kwiet kwiet sierp sierp kwiet kwiet Berliner sierp sierp sierp kwiet kwiet kwiet sierp kwiet sierp kwiet sierp kwiet kwiet sierp kwiet kwiet kwiet kwiet kwiet\x91 kwiet\x91 kwiet sierp kwiet sierp sierp kwiet\x91zil\x91 kwiet\x91zil sierp kwiet sierp sierp kwiet kwiet\x91 kwiet kwiet kwiet kwiet kwiet kwiet sierp∙ kwiet kwiet\x91 sierp kwiet Berliner sierp paździer sierp kwiet kwiet sierp∙ kwiet kwiet kwiet kwiet kwiet kwiet kwiet kwiet sierp\x91 sierp sierp sierp kwiet kwiet kwiet kwiet kwiet sierp sierp kwiet kwiet sierp sierp sierp sierp kwiet Berliner sierp sierp kwiet kwiet sierp sierp sierp\x91 kwiet sierp kwiet sierp kwiet sierp kwiet kwiet kwiet kwiet kwiet kwiet sierp sierp kwiet kwiet kwiet kwiet sierp sierp kwiet kwiet sierp kwiet∙ kwiet kwiet kwiet kwiet kwiet sierp sierp∙ sierp kwiet kwiet\x91 sierp sierp kwiet sierp\x91 kwiet sierp sierp kwiet sierp Berliner kwietzil sierp sierp sierp kwiet kwietnut sierp sierp sierp\x91 kwietnut sierp sierp sierp kwiet sierp kwiet kwiet kwiet sierp kwiet kwiet kwiet Berliner kwiet sierp kwiet sierp kwietzil\x91 sierpzil kwietnut kwiet kwiet sierp kwiet kwietútBrain sierpnut kwietnut sierpfenfen� kwiet kwiet kwiet\x91 sierp Berliner kwiet sierpnutardi kwietfen kwiet kwiet kwietút kwiet sierpnut sierpzilnutfennutútnut kwiet sierp kwiettreesfen�nut\x91 sierp sierpnut kwiet kwiet kwiet everybody sierp kwietnut kwiet sierp kwiet kwiet sierpnut sierp sierpnut sierp подаnutnut kwietlanguageʹnut∙fennut Berlinerfennutnutnutnutfoundnutfen kwietnutnut kwiet kwietnut�nut� sierpfen sierp sierp райnut kwietnut kwiet kwiet�nutnut kwietfenfenVol Einsfen kwiet sierp everybodyfen sierp kwiet sierpfentreesfenfennutzil sierpútfenhuifen sierpemfen kwietdu sierpfennutnutzilziltreesnutfen kwiet sierpfen BerlineremhuifenzilnutnutfennutfenBrainfenfentreesnutnuttreesnut Einsnutfennutnutcomútfennutfenzig‒zigzilfenzigfenfenfenfenfenfenfenemBrainfenfenzilfenʹfennut─fenfenfenfenfennuttreesuknutvoltreesfenfenútútfenemfenfenfenfenfenvoltreesfenfenfenfenhuibotfenfenfen¯nutfenfenzigспоfenfenútfennutfenfenfenfenhuifenfenfenfenútfenfenfenfentreesfenVolVolfenfenbotfenʹnutútzigспоfenziluzfentrees伝Volnutfenfriedfenfenfenfenfenfranfen uploadtreesʹemútnutfenfenem volunte¯uznuthuiemzigzilfen BerlinerfriedfenzigfenfenútfenfenuzFSfeneedúntreesúttreesfenfenfenfeneedTuTuFS BayernTuzigfenTufenfeneedfenfenzigcludfranútfenuzTufenitementVolfenuzfenTufenfenfenदFSfenúthuiTuTutreesútTufenFSfenfenFSfenfenfenútʹfenfenfenúnfennutfenfenּfenʹfenarishuifried fsспоदhuitzfeneedfenfenfeneedemVolnutTufenद voemfenFSeedeedFSún Berliner voeedfenvolfenMBjenFSTunutFShui vofennut VolDrageldVolhuiFSTunut voTu\x07FSчин voFSFS vobuchhuiTuFSFSFSआFSTuUMzigFSMBFS voluntevanvol vo voTu BerlinerjenFSfenFSjenTuTuvaneldFSjenhuiFSFS tu voUMumFSFSFSFSFSTuFSFSFSfenchenआ\x07FSFSFSFSFSFSFSFSFSeedFSFSआvanFSFS\x07FSFShuiFSeldFSFSFSTuFS voआjenFSFSFSFSFSFSFSjenjeneed vojenjenFSTufenumeedjen tuFSFSFSFSUMMBFSedu voluntefeneedFSFSFSFSFSFSFSFSeedFSFSFSFSeedFSFSFSchenFSFSFSFSFSFSчинFSjeneedMBchenFS\x07\x07aultumjenMBMBjenchencheneldUMchenMBFSMBchenFSFSFSjenFSeedeedeldFSFSFSeedBASEMBjeneedjeneedeedTuMBjenFSjen volunteFSchenhistoryFSeed voMB arbeitMBeedFSFSazjenआanterFSFSFS voluntechenFS henjeneldjenazjenariseedeldeldaulteedjenjeneldआпрачиjeneldFSFSFSeldeldMBjenMB\x07 AußereldMBeldMBedujenhapsault'}]
[2023-10-12 05:19:06,129] [INFO] [launch.py:347:main] Process 64168 exits successfully.
[2023-10-12 05:19:06,130] [INFO] [launch.py:347:main] Process 64173 exits successfully.
[2023-10-12 05:19:06,130] [INFO] [launch.py:347:main] Process 64170 exits successfully.
[2023-10-12 05:19:06,130] [INFO] [launch.py:347:main] Process 64166 exits successfully.

Start with script.py, the DeepSpeed inference tutorial, and the calls to deepspeed.init_inference and generate(). Reproduce the four-GPU run with the supplied nsys command, then inspect the NCCL timing and behavior when max_new_tokens is above 1024. Done means documenting the confirmed cause or a clearly reproducible DeepSpeed bug with supporting profiler output.