[BUG]AssertionError: Client Optimizer (type = <class 'NoneType'> is not instantiated but Client LR Scheduler is instantiated

root@mz32-01:/home/amd01/llm_dev/DeepSpeedExamples-megatron-LM/Megatron-LM# bash scripts/ds_zero-offload_pretrain_gpt2_model_parallel.sh
deepspeed --num_nodes 1 --num_gpus 2 pretrain_gpt2.py --model-parallel-size 2 --num-layers 4 --hidden-size 1024 --num-attention-heads 16 --batch-size 12 --seq-length 1024 --max-position-embeddings 1024 --train-iters 100 --resume-dataloader --train-data webtext --lazy-loader --tokenizer-type GPT2BPETokenizer --split 400,300,300 --distributed-backend nccl --lr 0.00015 --no-load-optim --lr-decay-style cosine --weight-decay 1e-2 --clip-grad 1.0 --warmup .01 --checkpoint-activations --deepspeed-activation-checkpointing --fp16 --log-interval 5 --deepspeed --deepspeed_config /home/amd01/llm_dev/DeepSpeedExamples-megatron-LM/Megatron-LM/scripts/ds_zero-offload_config.json
[2023-07-15 13:08:56,946] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-15 13:08:57,864] [WARNING] [runner.py:196:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2023-07-15 13:08:57,879] [INFO] [runner.py:555:main] cmd = /usr/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMV19 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None pretrain_gpt2.py --model-parallel-size 2 --num-layers 4 --hidden-size 1024 --num-attention-heads 16 --batch-size 12 --seq-length 1024 --max-position-embeddings 1024 --train-iters 100 --resume-dataloader --train-data webtext --lazy-loader --tokenizer-type GPT2BPETokenizer --split 400,300,300 --distributed-backend nccl --lr 0.00015 --no-load-optim --lr-decay-style cosine --weight-decay 1e-2 --clip-grad 1.0 --warmup .01 --checkpoint-activations --deepspeed-activation-checkpointing --fp16 --log-interval 5 --deepspeed --deepspeed_config /home/amd01/llm_dev/DeepSpeedExamples-megatron-LM/Megatron-LM/scripts/ds_zero-offload_config.json
[2023-07-15 13:08:58,758] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-15 13:08:59,632] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2023-07-15 13:08:59,632] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=2, node_rank=0
[2023-07-15 13:08:59,632] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
[2023-07-15 13:08:59,632] [INFO] [launch.py:163:main] dist_world_size=2
[2023-07-15 13:08:59,632] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0,1
[2023-07-15 13:09:00,499] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-15 13:09:00,523] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
using world size: 2 and model-parallel size: 2

using dynamic loss scaling
[2023-07-15 13:09:01,927] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2023-07-15 13:09:01,928] [INFO] [comm.py:594:init_distributed] cdb=None
[2023-07-15 13:09:01,928] [INFO] [comm.py:625:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2023-07-15 13:09:01,928] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2023-07-15 13:09:01,928] [INFO] [comm.py:594:init_distributed] cdb=None
initializing model parallel with size 2
[2023-07-15 13:09:01,971] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead
[2023-07-15 13:09:01,972] [INFO] [checkpointing.py:764:_configure_using_config_file] {'partition_activations': True, 'contiguous_memory_optimization': True, 'cpu_checkpointing': True, 'number_checkpoints': None, 'synchronize_checkpoint_boundary': False, 'profile': False}
Pretrain GPT2 model
arguments:
pretrained_bert .............. False
attention_dropout ............ 0.1
num_attention_heads .......... 16
hidden_size .................. 1024
intermediate_size ............ None
num_layers ................... 4
layernorm_epsilon ............ 1e-05
hidden_dropout ............... 0.1
max_position_embeddings ...... 1024
vocab_size ................... 30522
deep_init .................... False
make_vocab_size_divisible_by . 128
cpu_optimizer ................ False
cpu_torch_adam ............... False
fp16 ......................... True
fp32_embedding ............... False
fp32_layernorm ............... False
fp32_tokentypes .............. False
fp32_allreduce ............... False
hysteresis ................... 2
loss_scale ................... None
loss_scale_window ............ 1000
min_scale .................... 1
batch_size ................... 12
weight_decay ................. 0.01
checkpoint_activations ....... True
checkpoint_num_layers ........ 1
deepspeed_activation_checkpointing True
clip_grad .................... 1.0
train_iters .................. 100
log_interval ................. 5
exit_interval ................ None
seed ......................... 1234
reset_position_ids ........... False
reset_attention_mask ......... False
lr_decay_iters ............... None
lr_decay_style ............... cosine
lr ........................... 0.00015
warmup ....................... 0.01
save ......................... None
save_interval ................ 5000
no_save_optim ................ False
no_save_rng .................. False
load ......................... None
no_load_optim ................ True
no_load_rng .................. False
finetune ..................... False
resume_dataloader ............ True
distributed_backend .......... nccl
local_rank ................... 0
eval_batch_size .............. None
eval_iters ................... 100
eval_interval ................ 1000
eval_seq_length .............. None
eval_max_preds_per_seq ....... None
overlapping_eval ............. 32
cloze_eval ................... False
eval_hf ...................... False
load_openai .................. False
temperature .................. 1.0
top_p ........................ 0.0
top_k ........................ 0
out_seq_length ............... 256
model_parallel_size .......... 2
shuffle ...................... False
train_data ................... ['webtext']
use_npy_data_loader .......... False
train_data_path ..............
val_data_path ................
test_data_path ...............
input_data_sizes_file ........ sizes.txt
delim ........................ ,
text_key ..................... sentence
eval_text_key ................ None
valid_data ................... None
split ........................ 400,300,300
test_data .................... None
lazy_loader .................. True
loose_json ................... False
presplit_sentences ........... False
num_workers .................. 2
tokenizer_model_type ......... bert-large-uncased
tokenizer_path ............... tokenizer.model
tokenizer_type ............... GPT2BPETokenizer
cache_dir .................... None
use_tfrecords ................ False
seq_length ................... 1024
max_preds_per_seq ............ None
deepspeed .................... True
deepspeed_config ............. /home/amd01/llm_dev/DeepSpeedExamples-megatron-LM/Megatron-LM/scripts/ds_zero-offload_config.json
deepscale .................... False
deepscale_config ............. None
deepspeed_mpi ................ False
cuda ......................... True
rank ......................... 0
world_size ................... 2
dynamic_loss_scale ........... True
[2023-07-15 13:09:01,972] [INFO] [checkpointing.py:226:model_parallel_cuda_manual_seed] > initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234
configuring data
[2023-07-15 13:09:01,980] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead
padded vocab (size: 50257) with 175 dummy tokens (new size: 50432)
found end-of-document token: 50256
building GPT2 model ...
number of parameters on model parallel rank 0: 52076544
number of parameters on model parallel rank 1: 52076544
Optimizer = FusedAdam
learning rate decaying cosine
DeepSpeed is enabled.
[2023-07-15 13:09:06,340] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.5, git-hash=unknown, git-branch=unknown
[2023-07-15 13:09:06,341] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead
Traceback (most recent call last):
File "pretrain_gpt2.py", line 713, in
main()
File "pretrain_gpt2.py", line 661, in main
model, optimizer, lr_scheduler = setup_model_and_optimizer(args)
File "pretrain_gpt2.py", line 167, in setup_model_and_optimizer
model, optimizer, _, lr_scheduler = deepspeed.initialize(
File "/usr/local/lib/python3.8/dist-packages/deepspeed/init.py", line 165, in initialize
engine = DeepSpeedEngine(args=args,
File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 245, in init
self._do_sanity_check()
File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 993, in _do_sanity_check
assert isinstance(self.client_optimizer, Optimizer),
AssertionError: Client Optimizer (type = <class 'NoneType'> is not instantiated but Client LR Scheduler is instantiated
Optimizer = FusedAdam
[2023-07-15 13:09:06,364] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter cpu_offload is deprecated use offload_optimizer instead
Traceback (most recent call last):
File "pretrain_gpt2.py", line 713, in
main()
File "pretrain_gpt2.py", line 661, in main
model, optimizer, lr_scheduler = setup_model_and_optimizer(args)
File "pretrain_gpt2.py", line 167, in setup_model_and_optimizer
model, optimizer, _, lr_scheduler = deepspeed.initialize(
File "/usr/local/lib/python3.8/dist-packages/deepspeed/init.py", line 165, in initialize
engine = DeepSpeedEngine(args=args,
File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 245, in init
self._do_sanity_check()
File "/usr/local/lib/python3.8/dist-packages/deepspeed/runtime/engine.py", line 993, in _do_sanity_check
assert isinstance(self.client_optimizer, Optimizer),
AssertionError: Client Optimizer (type = <class 'NoneType'> is not instantiated but Client LR Scheduler is instantiated
[2023-07-15 13:09:07,656] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 2916
[2023-07-15 13:09:07,666] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 2917
[2023-07-15 13:09:07,666] [ERROR] [launch.py:321:sigkill_handler] ['/usr/bin/python3', '-u', 'pretrain_gpt2.py', '--local_rank=1', '--model-parallel-size', '2', '--num-layers', '4', '--hidden-size', '1024', '--num-attention-heads', '16', '--batch-size', '12', '--seq-length', '1024', '--max-position-embeddings', '1024', '--train-iters', '100', '--resume-dataloader', '--train-data', 'webtext', '--lazy-loader', '--tokenizer-type', 'GPT2BPETokenizer', '--split', '400,300,300', '--distributed-backend', 'nccl', '--lr', '0.00015', '--no-load-optim', '--lr-decay-style', 'cosine', '--weight-decay', '1e-2', '--clip-grad', '1.0', '--warmup', '.01', '--checkpoint-activations', '--deepspeed-activation-checkpointing', '--fp16', '--log-interval', '5', '--deepspeed', '--deepspeed_config', '/home/amd01/llm_dev/DeepSpeedExamples-megatron-LM/Megatron-LM/scripts/ds_zero-offload_config.json'] exits with return code = 1
root@mz32-01:/home/amd01/llm_dev/DeepSpeedExamples-megatron-LM/Megatron-LM#

Start by reproducing the command from scripts/ds_zero-offload_pretrain_gpt2_model_parallel.sh with scripts/ds_zero-offload_config.json. Read pretrain_gpt2.py around setup_model_and_optimizer and then DeepSpeed's runtime/engine.py around _do_sanity_check to trace the optimizer and scheduler inputs. Done means the reported configuration initializes without the assertion and the training command proceeds.