deepspeedai / deepspeedai/DeepSpeed

[BUG]deepspeed+llama factory realizes the case of connection interruption in single multi-card fine-tuning and the need for amazing video memory

Open
#5,222 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug compression
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Describe the bug
deepspeed+llama factory realizes the case of connection interruption in single multi-card fine-tuning and the need for amazing video memory

To Reproduce

  1. ubuntu with wsl2 with 2×4090 24G
  2. llama factory repo
  3. run the fine-tuning script

my script as follows：

export NCCL_P2P_DISABLE=1
export NCCL_IB_DISABLE=1
deepspeed --num_gpus 2 --master_port 6666 src/train_bash.py \
    --deepspeed ds_config.json \
    --stage sft \
    --do_train True \
    --model_name_or_path /mnt/wsl/PHYSICALDRIVE2p1/Baichuan2/baichuan-inc/Baichuan2-7B-Chat \
    --dataset_dir /mnt/wsl/PHYSICALDRIVE2p1/dataset \
    --dataset follow_up_self_cognition,shibing624_medical_finetune,alpaca_gpt4_en,alpaca_gpt4_zh \
    --template baichuan2 \
    --finetuning_type lora \
    --cutoff_len 1024 \
    --learning_rate 0.0001 \
    --num_train_epochs 3.0 \
    --max_samples 10000 \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 4 \
    --lr_scheduler_type cosine \
    --max_grad_norm 1.0 \
    --logging_steps 10 \
    --save_steps 100 \
    --warmup_steps 0 \
    --output_dir saves/Baichuan2-7B-Chat/lora/baichuan_medical \
    --bf16 True \
    --lora_rank 8 \
    --lora_dropout 0.1 \
    --lora_target W_pack \
    --plot_loss True

my ds_config.json as follows:

{
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu" :"auto",
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "bf16": {
        "enabled": true
    },
    "zero_optimization": {
    "stage": 3,
    "overlap_comm": true,
    "stage3_gather_16bit_weights_on_model_save": true
    },
    "flops_profiler": {
        "enabled": false,
        "profile_step": 1,
        "module_depth": -1,
        "top_modules": 1,
        "detailed": true,
        "output_file": null
    }
}

ERROR LOG

03/04/2024 12:04:07 - INFO - llmtuner.data.loader - Loading dataset shibing624_medical_finetune_organ.json...
03/04/2024 12:08:56 - INFO - llmtuner.data.loader - Loading dataset alpaca_gpt4_data_en.json...
03/04/2024 12:08:58 - INFO - llmtuner.data.loader - Loading dataset alpaca_gpt4_data_zh.json...
[INFO|training_args.py:1902] 2024-03-04 12:08:59,829 >> PyTorch: setting up devices
/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/transformers/training_args.py:1815: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead.
  warnings.warn(
/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/transformers/training_args.py:1815: FutureWarning: `--push_to_hub_token` is deprecated and will be removed in version 5 of 🤗 Transformers. Use `--hub_token` instead.
  warnings.warn(
[INFO|trainer.py:601] 2024-03-04 12:09:44,844 >> Using auto half precision backend
[2024-03-04 12:09:46,110] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.13.4, git-hash=unknown, git-branch=unknown
[2024-03-04 12:09:46,145] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2024-03-04 12:09:46,152] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2024-03-04 12:09:46,153] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2024-03-04 12:09:46,162] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = AdamW
[2024-03-04 12:09:46,162] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=AdamW type=<class 'torch.optim.adamw.AdamW'>
[2024-03-04 12:09:46,162] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False
[2024-03-04 12:09:46,162] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer
[2024-03-04 12:09:47,131] [INFO] [utils.py:800:see_memory_usage] Stage 3 initialize beginning
[2024-03-04 12:09:47,132] [INFO] [utils.py:801:see_memory_usage] MA 7.21 GB         Max_MA 9.12 GB         CA 7.25 GB         Max_CA 9 GB 
[2024-03-04 12:09:47,133] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory:  used = 10.04 GB, percent = 20.5%
[2024-03-04 12:09:47,151] [INFO] [stage3.py:130:__init__] Reduce bucket size 500,000,000
[2024-03-04 12:09:47,151] [INFO] [stage3.py:131:__init__] Prefetch bucket size 50,000,000
[2024-03-04 12:09:48,065] [INFO] [utils.py:800:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]
[2024-03-04 12:09:48,067] [INFO] [utils.py:801:see_memory_usage] MA 7.21 GB         Max_MA 7.21 GB         CA 7.25 GB         Max_CA 7 GB 
[2024-03-04 12:09:48,067] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory:  used = 10.04 GB, percent = 20.5%
Parameter Offload: Total persistent parameters: 4460544 in 129 params
[2024-03-04 12:09:48,767] [INFO] [utils.py:800:see_memory_usage] DeepSpeedZeRoOffload initialize [end]
[2024-03-04 12:09:48,769] [INFO] [utils.py:801:see_memory_usage] MA 7.2 GB         Max_MA 7.21 GB         CA 7.25 GB         Max_CA 7 GB 
[2024-03-04 12:09:48,769] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory:  used = 10.04 GB, percent = 20.5%
[2024-03-04 12:09:49,190] [INFO] [utils.py:800:see_memory_usage] Before creating fp16 partitions
[2024-03-04 12:09:49,191] [INFO] [utils.py:801:see_memory_usage] MA 7.2 GB         Max_MA 7.2 GB         CA 7.25 GB         Max_CA 7 GB 
[2024-03-04 12:09:49,192] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory:  used = 10.04 GB, percent = 20.5%
[2024-03-04 12:09:50,196] [INFO] [utils.py:800:see_memory_usage] After creating fp16 partitions: 1
[2024-03-04 12:09:50,199] [INFO] [utils.py:801:see_memory_usage] MA 7.2 GB         Max_MA 7.2 GB         CA 7.24 GB         Max_CA 7 GB 
[2024-03-04 12:09:50,199] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory:  used = 10.04 GB, percent = 20.5%
[2024-03-04 12:09:50,591] [INFO] [utils.py:800:see_memory_usage] Before creating fp32 partitions
[2024-03-04 12:09:50,593] [INFO] [utils.py:801:see_memory_usage] MA 7.2 GB         Max_MA 7.2 GB         CA 7.24 GB         Max_CA 7 GB 
[2024-03-04 12:09:50,593] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory:  used = 10.04 GB, percent = 20.5%
[2024-03-04 12:09:51,150] [INFO] [utils.py:800:see_memory_usage] After creating fp32 partitions
[2024-03-04 12:09:51,152] [INFO] [utils.py:801:see_memory_usage] MA 7.21 GB         Max_MA 7.22 GB         CA 7.24 GB         Max_CA 7 GB 
[2024-03-04 12:09:51,153] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory:  used = 10.04 GB, percent = 20.5%
[2024-03-04 12:09:52,088] [INFO] [utils.py:800:see_memory_usage] Before initializing optimizer states
[2024-03-04 12:09:52,090] [INFO] [utils.py:801:see_memory_usage] MA 7.21 GB         Max_MA 7.21 GB         CA 7.24 GB         Max_CA 7 GB 
[2024-03-04 12:09:52,092] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory:  used = 10.04 GB, percent = 20.5%
[2024-03-04 12:09:52,907] [INFO] [utils.py:800:see_memory_usage] After initializing optimizer states
[2024-03-04 12:09:52,909] [INFO] [utils.py:801:see_memory_usage] MA 7.21 GB         Max_MA 7.22 GB         CA 7.24 GB         Max_CA 7 GB 
[2024-03-04 12:09:52,910] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory:  used = 10.05 GB, percent = 20.5%
[2024-03-04 12:09:52,912] [INFO] [stage3.py:486:_setup_for_real_optimizer] optimizer state initialized
[2024-03-04 12:09:53,726] [INFO] [utils.py:800:see_memory_usage] After initializing ZeRO optimizer
[2024-03-04 12:09:53,728] [INFO] [utils.py:801:see_memory_usage] MA 8.15 GB         Max_MA 8.15 GB         CA 8.17 GB         Max_CA 8 GB 
[2024-03-04 12:09:53,728] [INFO] [utils.py:808:see_memory_usage] CPU Virtual Memory:  used = 10.05 GB, percent = 20.5%
[2024-03-04 12:09:53,729] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = AdamW
[2024-03-04 12:09:53,729] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client LR scheduler
[2024-03-04 12:09:53,729] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = None
[2024-03-04 12:09:53,729] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0001], mom=[(0.9, 0.999)]
[2024-03-04 12:09:53,733] [INFO] [config.py:996:print] DeepSpeedEngine configuration:
[2024-03-04 12:09:53,733] [INFO] [config.py:1000:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2024-03-04 12:09:53,733] [INFO] [config.py:1000:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2024-03-04 12:09:53,733] [INFO] [config.py:1000:print]   amp_enabled .................. False
[2024-03-04 12:09:53,734] [INFO] [config.py:1000:print]   amp_params ................... False
[2024-03-04 12:09:53,734] [INFO] [config.py:1000:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2024-03-04 12:09:53,734] [INFO] [config.py:1000:print]   bfloat16_enabled ............. True
[2024-03-04 12:09:53,734] [INFO] [config.py:1000:print]   bfloat16_immediate_grad_update  False
[2024-03-04 12:09:53,734] [INFO] [config.py:1000:print]   checkpoint_parallel_write_pipeline  False
[2024-03-04 12:09:53,734] [INFO] [config.py:1000:print]   checkpoint_tag_validation_enabled  True
[2024-03-04 12:09:53,734] [INFO] [config.py:1000:print]   checkpoint_tag_validation_fail  False
[2024-03-04 12:09:53,735] [INFO] [config.py:1000:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f94b0722620>
[2024-03-04 12:09:53,735] [INFO] [config.py:1000:print]   communication_data_type ...... None
[2024-03-04 12:09:53,735] [INFO] [config.py:1000:print]   compile_config ............... enabled=False backend='inductor' kwargs={}
[2024-03-04 12:09:53,735] [INFO] [config.py:1000:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2024-03-04 12:09:53,735] [INFO] [config.py:1000:print]   curriculum_enabled_legacy .... False
[2024-03-04 12:09:53,735] [INFO] [config.py:1000:print]   curriculum_params_legacy ..... False
[2024-03-04 12:09:53,735] [INFO] [config.py:1000:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2024-03-04 12:09:53,736] [INFO] [config.py:1000:print]   data_efficiency_enabled ...... False
[2024-03-04 12:09:53,736] [INFO] [config.py:1000:print]   dataloader_drop_last ......... False
[2024-03-04 12:09:53,736] [INFO] [config.py:1000:print]   disable_allgather ............ False
[2024-03-04 12:09:53,736] [INFO] [config.py:1000:print]   dump_state ................... False
[2024-03-04 12:09:53,736] [INFO] [config.py:1000:print]   dynamic_loss_scale_args ...... None
[2024-03-04 12:09:53,736] [INFO] [config.py:1000:print]   eigenvalue_enabled ........... False
[2024-03-04 12:09:53,736] [INFO] [config.py:1000:print]   eigenvalue_gas_boundary_resolution  1
[2024-03-04 12:09:53,736] [INFO] [config.py:1000:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2024-03-04 12:09:53,736] [INFO] [config.py:1000:print]   eigenvalue_layer_num ......... 0
[2024-03-04 12:09:53,736] [INFO] [config.py:1000:print]   eigenvalue_max_iter .......... 100
[2024-03-04 12:09:53,737] [INFO] [config.py:1000:print]   eigenvalue_stability ......... 1e-06
[2024-03-04 12:09:53,737] [INFO] [config.py:1000:print]   eigenvalue_tol ............... 0.01
[2024-03-04 12:09:53,737] [INFO] [config.py:1000:print]   eigenvalue_verbose ........... False
[2024-03-04 12:09:53,737] [INFO] [config.py:1000:print]   elasticity_enabled ........... False
[2024-03-04 12:09:53,737] [INFO] [config.py:1000:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2024-03-04 12:09:53,737] [INFO] [config.py:1000:print]   fp16_auto_cast ............... None
[2024-03-04 12:09:53,738] [INFO] [config.py:1000:print]   fp16_enabled ................. False
[2024-03-04 12:09:53,738] [INFO] [config.py:1000:print]   fp16_master_weights_and_gradients  False
[2024-03-04 12:09:53,741] [INFO] [config.py:1000:print]   global_rank .................. 0
[2024-03-04 12:09:53,741] [INFO] [config.py:1000:print]   grad_accum_dtype ............. None
[2024-03-04 12:09:53,741] [INFO] [config.py:1000:print]   gradient_accumulation_steps .. 4
[2024-03-04 12:09:53,741] [INFO] [config.py:1000:print]   gradient_clipping ............ 1.0
[2024-03-04 12:09:53,741] [INFO] [config.py:1000:print]   gradient_predivide_factor .... 1.0
[2024-03-04 12:09:53,741] [INFO] [config.py:1000:print]   graph_harvesting ............. False
[2024-03-04 12:09:53,742] [INFO] [config.py:1000:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2024-03-04 12:09:53,742] [INFO] [config.py:1000:print]   initial_dynamic_scale ........ 1
[2024-03-04 12:09:53,742] [INFO] [config.py:1000:print]   load_universal_checkpoint .... False
[2024-03-04 12:09:53,742] [INFO] [config.py:1000:print]   loss_scale ................... 1.0
[2024-03-04 12:09:53,742] [INFO] [config.py:1000:print]   memory_breakdown ............. False
[2024-03-04 12:09:53,742] [INFO] [config.py:1000:print]   mics_hierarchial_params_gather  False
[2024-03-04 12:09:53,742] [INFO] [config.py:1000:print]   mics_shard_size .............. -1
[2024-03-04 12:09:53,742] [INFO] [config.py:1000:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False
[2024-03-04 12:09:53,743] [INFO] [config.py:1000:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2024-03-04 12:09:53,743] [INFO] [config.py:1000:print]   optimizer_legacy_fusion ...... False
[2024-03-04 12:09:53,743] [INFO] [config.py:1000:print]   optimizer_name ............... None
[2024-03-04 12:09:53,743] [INFO] [config.py:1000:print]   optimizer_params ............. None
[2024-03-04 12:09:53,743] [INFO] [config.py:1000:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2024-03-04 12:09:53,743] [INFO] [config.py:1000:print]   pld_enabled .................. False
[2024-03-04 12:09:53,743] [INFO] [config.py:1000:print]   pld_params ................... False
[2024-03-04 12:09:53,743] [INFO] [config.py:1000:print]   prescale_gradients ........... False
[2024-03-04 12:09:53,743] [INFO] [config.py:1000:print]   scheduler_name ............... None
[2024-03-04 12:09:53,743] [INFO] [config.py:1000:print]   scheduler_params ............. None
[2024-03-04 12:09:53,743] [INFO] [config.py:1000:print]   seq_parallel_communication_data_type  torch.float32
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   sparse_attention ............. None
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   sparse_gradients_enabled ..... False
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   steps_per_print .............. inf
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   train_batch_size ............. 16
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   train_micro_batch_size_per_gpu  2
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   use_data_before_expert_parallel_  False
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   use_node_local_storage ....... False
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   wall_clock_breakdown ......... False
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   weight_quantization_config ... None
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   world_size ................... 2
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   zero_allow_untested_optimizer  True
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500,000,000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=True stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   zero_enabled ................. True
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   zero_force_ds_cpu_optimizer .. True
[2024-03-04 12:09:53,744] [INFO] [config.py:1000:print]   zero_optimization_stage ...... 3
[2024-03-04 12:09:53,745] [INFO] [config.py:986:print_user_config]   json = {
    "train_batch_size": 16, 
    "train_micro_batch_size_per_gpu": 2, 
    "gradient_accumulation_steps": 4, 
    "gradient_clipping": 1.0, 
    "bf16": {
        "enabled": true
    }, 
    "zero_optimization": {
        "stage": 3, 
        "overlap_comm": true, 
        "stage3_gather_16bit_weights_on_model_save": true
    }, 
    "flops_profiler": {
        "enabled": false, 
        "profile_step": 1, 
        "module_depth": -1, 
        "top_modules": 1, 
        "detailed": true, 
        "output_file": null
    }, 
    "steps_per_print": inf, 
    "fp16": {
        "enabled": false
    }, 
    "zero_allow_untested_optimizer": true
}
[INFO|trainer.py:1812] 2024-03-04 12:09:53,746 >> ***** Running training *****
[INFO|trainer.py:1813] 2024-03-04 12:09:53,746 >>   Num examples = 30,080
[INFO|trainer.py:1814] 2024-03-04 12:09:53,746 >>   Num Epochs = 3
[INFO|trainer.py:1815] 2024-03-04 12:09:53,746 >>   Instantaneous batch size per device = 2
[INFO|trainer.py:1818] 2024-03-04 12:09:53,746 >>   Total train batch size (w. parallel, distributed & accumulation) = 16
[INFO|trainer.py:1819] 2024-03-04 12:09:53,746 >>   Gradient Accumulation steps = 4
[INFO|trainer.py:1820] 2024-03-04 12:09:53,746 >>   Total optimization steps = 5,640
[INFO|trainer.py:1821] 2024-03-04 12:09:53,750 >>   Number of trainable parameters = 4,194,304
  0%|                                                                                                                                 | 0/5640 [00:00<?, ?it/s]/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/torch/utils/checkpoint.py:460: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/torch/utils/checkpoint.py:460: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
Traceback (most recent call last):
  File "/home/mizzle/LLM/LLaMA-Factory/src/train_bash.py", line 14, in <module>
    main()
  File "/home/mizzle/LLM/LLaMA-Factory/src/train_bash.py", line 5, in main
    run_exp()
  File "/home/mizzle/LLM/LLaMA-Factory/src/llmtuner/train/tuner.py", line 31, in run_exp
    run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
  File "/home/mizzle/LLM/LLaMA-Factory/src/llmtuner/train/sft/workflow.py", line 75, in run_sft
    train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/transformers/trainer.py", line 1624, in train
    return inner_training_loop(
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/transformers/trainer.py", line 1961, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/transformers/trainer.py", line 2902, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/transformers/trainer.py", line 2925, in compute_loss
    outputs = model(**inputs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1852, in forward
    loss = self.module(*inputs, **kwargs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/peft/peft_model.py", line 1083, in forward
    return self.base_model(
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 161, in forward
    return self.model.forward(*args, **kwargs)
  File "/home/mizzle/.cache/huggingface/modules/transformers_modules/Baichuan2-7B-Chat/modeling_baichuan.py", line 713, in forward
    loss = loss_fct(shift_logits, shift_labels) + z_loss
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/torch/nn/modules/loss.py", line 1179, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "/home/mizzle/miniconda3/envs/ft/lib/python3.10/site-packages/torch/nn/functional.py", line 3059, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 176.00 MiB. GPU 0 has a total capacity of 23.99 GiB of which 7.24 GiB is free. Process 1933724 has 17179869184.00 GiB memory in use. Including non-PyTorch memory, this process has 17179869184.00 GiB memory in use. Of the allocated memory 12.52 GiB is allocated by PyTorch, and 1.54 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
  0%|                                                                                                                                 | 0/5640 [11:40<?, ?it/s]
[2024-03-04 12:21:46,972] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 1933723
[2024-03-04 12:21:46,973] [INFO] [launch.py:316:sigkill_handler] Killing subprocess 1933724

17179869184.00 GiB memory is too amazing!!!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named. Start by reproducing the supplied DeepSpeed command on WSL2 with two 24 GB RTX 4090 cards, then capture the complete error after the truncated configuration log and compare the ZeRO stage 3 initialization and memory behavior. Done means identifying whether the interruption or memory requirement is caused by DeepSpeed and documenting a verified fix or configuration change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.