deepspeedai / deepspeedai/DeepSpeed

Pipeline parallelism got stuck after switching off `fp16`

Open
#1,283 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Hi,

We're currently conducting experiments based on this example. We are able to run the script without problem with --fp16. However, the script seems to be stuck when --fp16 is commented out and "fp16 enabled" is switched to false here.

To reproduce the issue, here are the arguments we used (the experiment was conducted on a single node with 4 RTX 2080Ti GPUs):

using world size: 4 and model-parallel size: 2 
using torch.float32 for parameters ...
-------------------- arguments --------------------
  adam_beta1 ...................... 0.9
  adam_beta2 ...................... 0.999
  adam_eps ........................ 1e-08
  adlr_autoresume ................. False
  adlr_autoresume_interval ........ 1000
  apply_query_key_layer_scaling ... False
  apply_residual_connection_post_layernorm  False
  attention_dropout ............... 0.1
  attention_softmax_in_fp32 ....... False
  batch_size ...................... 4
  bert_load ....................... None
  bias_dropout_fusion ............. False
  bias_gelu_fusion ................ False
  block_data_path ................. None
  checkpoint_activations .......... True
  checkpoint_in_cpu ............... False
  checkpoint_num_layers ........... 1
  clip_grad ....................... 1.0
  contigious_checkpointing ........ True
  cpu_optimizer ................... False
  cpu_torch_adam .................. False
  data_impl ....................... mmap
  data_path ....................... /workspace/data/my-gpt2_text_document/my-gpt2_text_document
  DDP_impl ........................ local
  deepscale ....................... False
  deepscale_config ................ None
  deepspeed ....................... True
  deepspeed_activation_checkpointing  False
  deepspeed_config ................ /workspace/DeepSpeed/DeepSpeedExamples/Megatron-LM-v1.1.5-3D_parallelism/examples/ds_config.json
  deepspeed_mpi ................... False
  distribute_checkpointed_activations  False
  distributed_backend ............. nccl
  dynamic_loss_scale .............. True
  eod_mask_loss ................... False
  eval_interval ................... 100
  eval_iters ...................... 10
  exit_interval ................... None
  faiss_use_gpu ................... False
  finetune ........................ False
  fp16 ............................ False
  fp16_lm_cross_entropy ........... False
  fp32_allreduce .................. False
  gas ............................. 16
  hidden_dropout .................. 0.1
  hidden_size ..................... 1024
  hysteresis ...................... 2
  ict_head_size ................... None
  ict_load ........................ None
  indexer_batch_size .............. 128
  indexer_log_interval ............ 1000
  init_method_std ................. 0.02
  layernorm_epsilon ............... 1e-05
  lazy_mpu_init ................... None
  load ............................ /workspace/checkpoints/gpt2_345m_ds
  local_rank ...................... 0
  log_interval .................... 1
  loss_scale ...................... None
  loss_scale_window ............... 1000
  lr .............................. 0.00015
  lr_decay_iters .................. 320000
  lr_decay_style .................. cosine
  make_vocab_size_divisible_by .... 128
  mask_prob ....................... 0.15
  max_position_embeddings ......... 1024
  merge_file ...................... /workspace/data/gpt2-merges.txt
  min_lr .......................... 1e-05
  min_scale ....................... 1
  mmap_warmup ..................... False
  model_parallel_size ............. 2
  no_load_optim ................... False
  no_load_rng ..................... False
  no_save_optim ................... False
  no_save_rng ..................... False
  num_attention_heads ............. 16
  num_layers ...................... 24
  num_unique_layers ............... None
  num_workers ..................... 2
  onnx_safe ....................... None
  openai_gelu ..................... False
  override_lr_scheduler ........... False
  param_sharing_style ............. grouped
  params_dtype .................... torch.float32
  partition_activations ........... True
  pipe_parallel_size .............. 2
  profile_backward ................ False
  query_in_block_prob ............. 0.1
  rank ............................ 0
  report_topk_accuracies .......... []
  reset_attention_mask ............ False
  reset_position_ids .............. False
  save ............................ /workspace/checkpoints/gpt2_345m_ds
  save_interval ................... 500
  scaled_masked_softmax_fusion .... False
  scaled_upper_triang_masked_softmax_fusion  False
  seed ............................ 1234
  seq_length ...................... 1024
  short_seq_prob .................. 0.1
  split ........................... 949,50,1
  synchronize_each_layer .......... True
  tensorboard_dir ................. tensorboard_data/24l_1024h_1n_4g_2pp_2mp_4b_ds4
  titles_data_path ................ None
  tokenizer_type .................. GPT2BPETokenizer
  train_iters ..................... 320000
  use_checkpoint_lr_scheduler ..... False
  use_cpu_initialization .......... False
  use_one_sent_docs ............... False
  vocab_file ...................... /workspace/data/gpt2-vocab.json
  warmup .......................... 0.01
  weight_decay .................... 0.01
  world_size ...................... 4
  zero_allgather_bucket_size ...... 5000000000
  zero_contigious_gradients ....... True
  zero_reduce_bucket_size ......... 50000000
  zero_reduce_scatter ............. True
  zero_stage ...................... 0
---------------- end of arguments ----------------

And the information from NCCL_DEBUG before the script got stuck:

training ...
9ea320dd7260:17364:17703 [0] NCCL INFO Channel 00/02 :    0   1
9ea320dd7260:17364:17703 [0] NCCL INFO Channel 01/02 :    0   1
9ea320dd7260:17365:17704 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
9ea320dd7260:17364:17703 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
9ea320dd7260:17364:17703 [0] NCCL INFO Setting affinity for GPU 0 to 0fffff
9ea320dd7260:17365:17704 [1] NCCL INFO Setting affinity for GPU 1 to 0fffff
9ea320dd7260:17364:17703 [0] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000)
9ea320dd7260:17365:17704 [1] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000)
9ea320dd7260:17364:17703 [0] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000)
9ea320dd7260:17364:17703 [0] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000)
9ea320dd7260:17364:17703 [0] NCCL INFO Channel 00 : 0[19000] -> 1[1a000] via direct shared memory
9ea320dd7260:17364:17703 [0] NCCL INFO Could not enable P2P between dev 0(=19000) and dev 1(=1a000)
9ea320dd7260:17365:17704 [1] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000)
9ea320dd7260:17364:17703 [0] NCCL INFO Channel 01 : 0[19000] -> 1[1a000] via direct shared memory
9ea320dd7260:17365:17704 [1] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000)
9ea320dd7260:17365:17704 [1] NCCL INFO Channel 00 : 1[1a000] -> 0[19000] via direct shared memory
9ea320dd7260:17365:17704 [1] NCCL INFO Could not enable P2P between dev 1(=1a000) and dev 0(=19000)
9ea320dd7260:17365:17704 [1] NCCL INFO Channel 01 : 1[1a000] -> 0[19000] via direct shared memory
9ea320dd7260:17365:17704 [1] NCCL INFO Connected all rings
9ea320dd7260:17365:17704 [1] NCCL INFO Connected all trees
9ea320dd7260:17365:17704 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
9ea320dd7260:17365:17704 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
9ea320dd7260:17365:17704 [1] NCCL INFO comm 0x7fe4600a08f0 rank 1 nranks 2 cudaDev 1 busId 1a000 - Init COMPLETE
9ea320dd7260:17364:17703 [0] NCCL INFO Connected all rings
9ea320dd7260:17364:17703 [0] NCCL INFO Connected all trees
9ea320dd7260:17364:17703 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
9ea320dd7260:17364:17703 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
9ea320dd7260:17364:17703 [0] NCCL INFO comm 0x7fb420008e10 rank 0 nranks 2 cudaDev 0 busId 19000 - Init COMPLETE
9ea320dd7260:17364:17364 [0] NCCL INFO Launch mode Parallel
[2021-08-04 03:42:57,432] [INFO] [checkpointing.py:400:forward] Activation Checkpointing Information
[2021-08-04 03:42:57,432] [INFO] [checkpointing.py:401:forward] ----Partition Activations False, CPU CHECKPOINTING False
[2021-08-04 03:42:57,432] [INFO] [checkpointing.py:404:forward] ----contiguous Memory Checkpointing False with None total layers
[2021-08-04 03:42:57,432] [INFO] [checkpointing.py:407:forward] ----Synchronization False
[2021-08-04 03:42:57,432] [INFO] [checkpointing.py:408:forward] ----Profiling time in checkpointing False
9ea320dd7260:17367:17720 [3] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
9ea320dd7260:17366:17719 [2] NCCL INFO Channel 00/02 :    0   1
9ea320dd7260:17366:17719 [2] NCCL INFO Channel 01/02 :    0   1
9ea320dd7260:17367:17720 [3] NCCL INFO Setting affinity for GPU 3 to 0fffff
9ea320dd7260:17366:17719 [2] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
9ea320dd7260:17366:17719 [2] NCCL INFO Setting affinity for GPU 2 to 0fffff
9ea320dd7260:17367:17720 [3] NCCL INFO Could not enable P2P between dev 3(=68000) and dev 2(=67000)
9ea320dd7260:17366:17719 [2] NCCL INFO Could not enable P2P between dev 2(=67000) and dev 3(=68000)
9ea320dd7260:17367:17720 [3] NCCL INFO Could not enable P2P between dev 3(=68000) and dev 2(=67000)
9ea320dd7260:17366:17719 [2] NCCL INFO Could not enable P2P between dev 2(=67000) and dev 3(=68000)
9ea320dd7260:17367:17720 [3] NCCL INFO Could not enable P2P between dev 3(=68000) and dev 2(=67000)
9ea320dd7260:17367:17720 [3] NCCL INFO Channel 00 : 1[68000] -> 0[67000] via direct shared memory
9ea320dd7260:17367:17720 [3] NCCL INFO Could not enable P2P between dev 3(=68000) and dev 2(=67000)
9ea320dd7260:17366:17719 [2] NCCL INFO Could not enable P2P between dev 2(=67000) and dev 3(=68000)
9ea320dd7260:17367:17720 [3] NCCL INFO Channel 01 : 1[68000] -> 0[67000] via direct shared memory
9ea320dd7260:17366:17719 [2] NCCL INFO Channel 00 : 0[67000] -> 1[68000] via direct shared memory
9ea320dd7260:17366:17719 [2] NCCL INFO Could not enable P2P between dev 2(=67000) and dev 3(=68000)
9ea320dd7260:17366:17719 [2] NCCL INFO Channel 01 : 0[67000] -> 1[68000] via direct shared memory
9ea320dd7260:17367:17720 [3] NCCL INFO Connected all rings
9ea320dd7260:17367:17720 [3] NCCL INFO Connected all trees
9ea320dd7260:17367:17720 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
9ea320dd7260:17367:17720 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
9ea320dd7260:17367:17720 [3] NCCL INFO comm 0x7fe0880a0670 rank 1 nranks 2 cudaDev 3 busId 68000 - Init COMPLETE
9ea320dd7260:17366:17719 [2] NCCL INFO Connected all rings
9ea320dd7260:17366:17719 [2] NCCL INFO Connected all trees
9ea320dd7260:17366:17719 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
9ea320dd7260:17366:17719 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
9ea320dd7260:17366:17719 [2] NCCL INFO comm 0x7fb274008e10 rank 0 nranks 2 cudaDev 2 busId 67000 - Init COMPLETE
9ea320dd7260:17366:17366 [2] NCCL INFO Launch mode Parallel
9ea320dd7260:17367:17727 [3] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
9ea320dd7260:17367:17727 [3] NCCL INFO Setting affinity for GPU 3 to 0fffff
9ea320dd7260:17366:17726 [2] NCCL INFO Channel 00/02 :    0   1
9ea320dd7260:17366:17726 [2] NCCL INFO Channel 01/02 :    0   1
9ea320dd7260:17366:17726 [2] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
9ea320dd7260:17366:17726 [2] NCCL INFO Setting affinity for GPU 2 to 0fffff
9ea320dd7260:17367:17727 [3] NCCL INFO Could not enable P2P between dev 3(=68000) and dev 2(=67000)
9ea320dd7260:17366:17726 [2] NCCL INFO Could not enable P2P between dev 2(=67000) and dev 3(=68000)
9ea320dd7260:17367:17727 [3] NCCL INFO Could not enable P2P between dev 3(=68000) and dev 2(=67000)
9ea320dd7260:17366:17726 [2] NCCL INFO Could not enable P2P between dev 2(=67000) and dev 3(=68000)
9ea320dd7260:17367:17727 [3] NCCL INFO Could not enable P2P between dev 3(=68000) and dev 2(=67000)
9ea320dd7260:17367:17727 [3] NCCL INFO Channel 00 : 1[68000] -> 0[67000] via direct shared memory
9ea320dd7260:17367:17727 [3] NCCL INFO Could not enable P2P between dev 3(=68000) and dev 2(=67000)
9ea320dd7260:17367:17727 [3] NCCL INFO Channel 01 : 1[68000] -> 0[67000] via direct shared memory
9ea320dd7260:17366:17726 [2] NCCL INFO Could not enable P2P between dev 2(=67000) and dev 3(=68000)
9ea320dd7260:17366:17726 [2] NCCL INFO Channel 00 : 0[67000] -> 1[68000] via direct shared memory
9ea320dd7260:17366:17726 [2] NCCL INFO Could not enable P2P between dev 2(=67000) and dev 3(=68000)
9ea320dd7260:17366:17726 [2] NCCL INFO Channel 01 : 0[67000] -> 1[68000] via direct shared memory
9ea320dd7260:17367:17727 [3] NCCL INFO Connected all rings
9ea320dd7260:17367:17727 [3] NCCL INFO Connected all trees
9ea320dd7260:17367:17727 [3] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
9ea320dd7260:17367:17727 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
9ea320dd7260:17367:17727 [3] NCCL INFO comm 0x7fe088272680 rank 1 nranks 2 cudaDev 3 busId 68000 - Init COMPLETE
9ea320dd7260:17366:17726 [2] NCCL INFO Connected all rings
9ea320dd7260:17366:17726 [2] NCCL INFO Connected all trees
9ea320dd7260:17366:17726 [2] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64
9ea320dd7260:17366:17726 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
9ea320dd7260:17366:17726 [2] NCCL INFO comm 0x7fb264008e10 rank 0 nranks 2 cudaDev 2 busId 67000 - Init COMPLETE
9ea320dd7260:17366:17366 [2] NCCL INFO Launch mode Parallel

And our environment is based on NGC docker image nvcr.io/nvidia/pytorch:20.12-py3.

torch                     1.8.0a0+1606899 
apex                      0.1
NVIDIA-SMI 450.51.05    Driver Version: 450.51.05    CUDA Version: 11.0

Any thoughts on why it happens? Thanks in advance!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/ds_pretrain_gpt2_pipe.sh and examples/ds_config.json, reproducing the four-GPU pipeline run with fp16 enabled and disabled. Compare the startup path and NCCL output for both configurations; done means the fp32 configuration completes without hanging. No source file or test is named in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.