deepspeedai / deepspeedai/DeepSpeed
[BUG] Training hits out of memory
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
I want to train llama2-7B with 2 nodes. Each node has 4 GPUs with 40 G cuda memory per GPU.
I'm using Torchrun to distribute training. Weight initialisation has been done, but when it go to training part, I got CUDA out of memory.
To Reproduce
Steps to reproduce the behavior:
- Go to '...'
- Click on '....'
- Scroll down to '....'
- See error
Expected behavior
It should be train without OOM.
ds_report output
[2023-11-22 11:18:38,344] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
DeepSpeed C++/CUDA extension op report
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
JIT compiled ops requires ninja
ninja .................. [OKAY]
op name ................ installed .. compatible
async_io ............... [YES] ...... [OKAY]
fused_adam ............. [YES] ...... [OKAY]
cpu_adam ............... [YES] ...... [OKAY]
cpu_adagrad ............ [YES] ...... [OKAY]
cpu_lion ............... [YES] ...... [OKAY]
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fused_lamb ............. [YES] ...... [OKAY]
fused_lion ............. [YES] ...... [OKAY]
inference_core_ops ..... [YES] ...... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
quantizer .............. [YES] ...... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [YES] ...... [OKAY]
random_ltd ............. [YES] ...... [OKAY]
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1
[WARNING] using untested triton version (2.1.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [YES] ...... [OKAY]
transformer ............ [YES] ...... [OKAY]
stochastic_transformer . [YES] ...... [OKAY]
transformer_inference .. [YES] ...... [OKAY]
Screenshots
If applicable, add screenshots to help explain your problem.
[2023-11-22 11:03:48,227] torch.distributed.run: [WARNING] master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.
[2023-11-22 11:03:48,247] torch.distributed.run: [WARNING] master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.
[2023-11-22 11:04:05,831] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-11-22 11:04:05,840] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-11-22 11:04:05,847] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-11-22 11:04:05,851] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-11-22 11:04:05,858] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-11-22 11:04:05,867] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-11-22 11:04:05,869] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-11-22 11:04:05,870] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
wandb: Currently logged in as: kasravi66 (chm-hci). Use wandb login --relogin to force relogin
wandb: Appending key for api.wandb.ai to your netrc file: /u/jkasravi/.netrc
wandb: Currently logged in as: kasravi66 (chm-hci). Use wandb login --relogin to force relogin
wandb: Currently logged in as: kasravi66 (chm-hci). Use wandb login --relogin to force relogin
wandb: Currently logged in as: kasravi66 (chm-hci). Use wandb login --relogin to force relogin
wandb: Appending key for api.wandb.ai to your netrc file: /u/jkasravi/.netrc
wandb: Appending key for api.wandb.ai to your netrc file: /u/jkasravi/.netrc
wandb: Appending key for api.wandb.ai to your netrc file: /u/jkasravi/.netrc
wandb: Currently logged in as: kasravi66 (chm-hci). Use wandb login --relogin to force relogin
wandb: Currently logged in as: kasravi66 (chm-hci). Use wandb login --relogin to force relogin
wandb: Appending key for api.wandb.ai to your netrc file: /u/jkasravi/.netrc
wandb: Appending key for api.wandb.ai to your netrc file: /u/jkasravi/.netrc
Loading config file from /llm_fine_tuning/src/tests/cluster/config/full.yml
{'fine_tuning_strategy': 'full_fine_tuning', 'unfrozen_layers': 10, 'epochs': 10, 'per_device_train_batch_size': 10, 'gradient_accumulation_steps': 10, 'gradient_checkpointing': True, 'log_on_each_node': False, 'seed': 0, 'data_seed': 0, 'tf32': True, 'output_dir': '/llm_fine_tuning/src/tests/cluster/output_dir', 'deepspeed_config': '/llm_fine_tuning/src/ds_config.json', 'model': 'llama27b', 'tokenized_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/tokenized/pilot_A_dev', 'chunks_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/chunks/pilot_A_dev', 'dataset': 'timdettmers/openassistant-guanaco', 'name': 'fine_tuning/algoinst_pilot_A/full', 'save_strategy': 'epoch', 'save_steps': 1, 'save_total_limit': 1}
[2023-11-22 11:04:54,928] [INFO] [comm.py:637:init_distributed] cdb=None
[2023-11-22 11:04:54,929] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
Loading config file from /llm_fine_tuning/src/tests/cluster/config/full.yml
{'fine_tuning_strategy': 'full_fine_tuning', 'unfrozen_layers': 10, 'epochs': 10, 'per_device_train_batch_size': 10, 'gradient_accumulation_steps': 10, 'gradient_checkpointing': True, 'log_on_each_node': False, 'seed': 0, 'data_seed': 0, 'tf32': True, 'output_dir': '/llm_fine_tuning/src/tests/cluster/output_dir', 'deepspeed_config': '/llm_fine_tuning/src/ds_config.json', 'model': 'llama27b', 'tokenized_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/tokenized/pilot_A_dev', 'chunks_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/chunks/pilot_A_dev', 'dataset': 'timdettmers/openassistant-guanaco', 'name': 'fine_tuning/algoinst_pilot_A/full', 'save_strategy': 'epoch', 'save_steps': 1, 'save_total_limit': 1}
[2023-11-22 11:04:55,268] [INFO] [comm.py:637:init_distributed] cdb=None
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py:671: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers. Please use token instead.
warnings.warn(
wandb: W&B API key is configured. Use wandb login --relogin to force relogin
wandb: Appending key for api.wandb.ai to your netrc file: /u/jkasravi/.netrc
wandb: W&B API key is configured. Use wandb login --relogin to force relogin
wandb: Appending key for api.wandb.ai to your netrc file: /u/jkasravi/.netrc
Loading config file from /llm_fine_tuning/src/tests/cluster/config/full.yml
{'fine_tuning_strategy': 'full_fine_tuning', 'unfrozen_layers': 10, 'epochs': 10, 'per_device_train_batch_size': 10, 'gradient_accumulation_steps': 10, 'gradient_checkpointing': True, 'log_on_each_node': False, 'seed': 0, 'data_seed': 0, 'tf32': True, 'output_dir': '/llm_fine_tuning/src/tests/cluster/output_dir', 'deepspeed_config': '/llm_fine_tuning/src/ds_config.json', 'model': 'llama27b', 'tokenized_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/tokenized/pilot_A_dev', 'chunks_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/chunks/pilot_A_dev', 'dataset': 'timdettmers/openassistant-guanaco', 'name': 'fine_tuning/algoinst_pilot_A/full', 'save_strategy': 'epoch', 'save_steps': 1, 'save_total_limit': 1}
[2023-11-22 11:04:56,803] [INFO] [comm.py:637:init_distributed] cdb=None
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py:671: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers. Please use token instead.
warnings.warn(
Loading config file from /llm_fine_tuning/src/tests/cluster/config/full.yml
{'fine_tuning_strategy': 'full_fine_tuning', 'unfrozen_layers': 10, 'epochs': 10, 'per_device_train_batch_size': 10, 'gradient_accumulation_steps': 10, 'gradient_checkpointing': True, 'log_on_each_node': False, 'seed': 0, 'data_seed': 0, 'tf32': True, 'output_dir': '/llm_fine_tuning/src/tests/cluster/output_dir', 'deepspeed_config': '/llm_fine_tuning/src/ds_config.json', 'model': 'llama27b', 'tokenized_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/tokenized/pilot_A_dev', 'chunks_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/chunks/pilot_A_dev', 'dataset': 'timdettmers/openassistant-guanaco', 'name': 'fine_tuning/algoinst_pilot_A/full', 'save_strategy': 'epoch', 'save_steps': 1, 'save_total_limit': 1}
[2023-11-22 11:04:59,539] [INFO] [comm.py:637:init_distributed] cdb=None
Loading config file from /llm_fine_tuning/src/tests/cluster/config/full.yml
{'fine_tuning_strategy': 'full_fine_tuning', 'unfrozen_layers': 10, 'epochs': 10, 'per_device_train_batch_size': 10, 'gradient_accumulation_steps': 10, 'gradient_checkpointing': True, 'log_on_each_node': False, 'seed': 0, 'data_seed': 0, 'tf32': True, 'output_dir': '/llm_fine_tuning/src/tests/cluster/output_dir', 'deepspeed_config': '/llm_fine_tuning/src/ds_config.json', 'model': 'llama27b', 'tokenized_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/tokenized/pilot_A_dev', 'chunks_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/chunks/pilot_A_dev', 'dataset': 'timdettmers/openassistant-guanaco', 'name': 'fine_tuning/algoinst_pilot_A/full', 'save_strategy': 'epoch', 'save_steps': 1, 'save_total_limit': 1}
[2023-11-22 11:04:59,547] [INFO] [comm.py:637:init_distributed] cdb=None
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py:671: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers. Please use token instead.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py:671: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers. Please use token instead.
warnings.warn(
Loading config file from /llm_fine_tuning/src/tests/cluster/config/full.yml
{'fine_tuning_strategy': 'full_fine_tuning', 'unfrozen_layers': 10, 'epochs': 10, 'per_device_train_batch_size': 10, 'gradient_accumulation_steps': 10, 'gradient_checkpointing': True, 'log_on_each_node': False, 'seed': 0, 'data_seed': 0, 'tf32': True, 'output_dir': '/llm_fine_tuning/src/tests/cluster/output_dir', 'deepspeed_config': '/llm_fine_tuning/src/ds_config.json', 'model': 'llama27b', 'tokenized_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/tokenized/pilot_A_dev', 'chunks_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/chunks/pilot_A_dev', 'dataset': 'timdettmers/openassistant-guanaco', 'name': 'fine_tuning/algoinst_pilot_A/full', 'save_strategy': 'epoch', 'save_steps': 1, 'save_total_limit': 1}
[2023-11-22 11:04:59,891] [INFO] [comm.py:637:init_distributed] cdb=None
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py:671: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers. Please use token instead.
warnings.warn(
Loading config file from /llm_fine_tuning/src/tests/cluster/config/full.yml
{'fine_tuning_strategy': 'full_fine_tuning', 'unfrozen_layers': 10, 'epochs': 10, 'per_device_train_batch_size': 10, 'gradient_accumulation_steps': 10, 'gradient_checkpointing': True, 'log_on_each_node': False, 'seed': 0, 'data_seed': 0, 'tf32': True, 'output_dir': '/llm_fine_tuning/src/tests/cluster/output_dir', 'deepspeed_config': '/llm_fine_tuning/src/ds_config.json', 'model': 'llama27b', 'tokenized_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/tokenized/pilot_A_dev', 'chunks_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/chunks/pilot_A_dev', 'dataset': 'timdettmers/openassistant-guanaco', 'name': 'fine_tuning/algoinst_pilot_A/full', 'save_strategy': 'epoch', 'save_steps': 1, 'save_total_limit': 1}
[2023-11-22 11:05:00,982] [INFO] [comm.py:637:init_distributed] cdb=None
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py:671: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers. Please use token instead.
warnings.warn(
Loading config file from /llm_fine_tuning/src/tests/cluster/config/full.yml
{'fine_tuning_strategy': 'full_fine_tuning', 'unfrozen_layers': 10, 'epochs': 10, 'per_device_train_batch_size': 10, 'gradient_accumulation_steps': 10, 'gradient_checkpointing': True, 'log_on_each_node': False, 'seed': 0, 'data_seed': 0, 'tf32': True, 'output_dir': '/llm_fine_tuning/src/tests/cluster/output_dir', 'deepspeed_config': '/llm_fine_tuning/src/ds_config.json', 'model': 'llama27b', 'tokenized_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/tokenized/pilot_A_dev', 'chunks_dataset': '/llm_fine_tuning/projects/algoinst/data_withou_lfs/chunks/pilot_A_dev', 'dataset': 'timdettmers/openassistant-guanaco', 'name': 'fine_tuning/algoinst_pilot_A/full', 'save_strategy': 'epoch', 'save_steps': 1, 'save_total_limit': 1}
[2023-11-22 11:05:01,417] [INFO] [comm.py:637:init_distributed] cdb=None
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py:671: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers. Please use token instead.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py:671: FutureWarning: The use_auth_token argument is deprecated and will be removed in v5 of Transformers. Please use token instead.
warnings.warn(
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:15<00:15, 15.76s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:21<00:00, 9.62s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:21<00:00, 10.54s/it]
model and tokenizer [LLAMA27BModel] were created
all params: 6,738,415,616 || trainable params: 6,738,415,616 || trainable%: 100.0
Loading checkpoint shards: 50%|█████ | 1/2 [00:14<00:14, 14.58s/it]
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:16<00:16, 16.07s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:23<00:00, 11.51s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:23<00:00, 11.97s/it]
model and tokenizer [LLAMA27BModel] were created
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 11.41s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:24<00:00, 12.11s/it]
model and tokenizer [LLAMA27BModel] were created
all params: 6,738,415,616 || trainable params: 6,738,415,616 || trainable%: 100.0
all params: 6,738,415,616 || trainable params: 6,738,415,616 || trainable%: 100.0
Saving the dataset (0/1 shards): 0%| | 0/8 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 303.27 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 301.46 examples/s]
Saving the dataset (0/1 shards): 0%| | 0/2 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 156.57 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 154.82 examples/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:11<00:11, 11.47s/it]
Saving the dataset (0/1 shards): 0%| | 0/8 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 344.53 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 342.07 examples/s]
Saving the dataset (0/1 shards): 0%| | 0/2 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 133.76 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 132.39 examples/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:14<00:00, 6.50s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:14<00:00, 7.25s/it]
Saving the dataset (0/1 shards): 0%| | 0/8 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 515.44 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 509.81 examples/s]
model and tokenizer [LLAMA27BModel] were created
all params: 6,738,415,616 || trainable params: 6,738,415,616 || trainable%: 100.0
Saving the dataset (0/1 shards): 0%| | 0/2 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 233.07 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 228.93 examples/s]
Saving the dataset (0/1 shards): 0%| | 0/8 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 525.96 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 519.98 examples/s]
Saving the dataset (0/1 shards): 0%| | 0/2 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 103.67 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 102.78 examples/s]
Training has been launched...
Training has been launched...
Training has been launched...
Training has been launched...
ravg1038:32091:32091 [0] NCCL INFO Bootstrap : Using ib0:10.181.140.38<0>
ravg1038:32093:32093 [2] NCCL INFO cudaDriverVersion 12020
ravg1038:32093:32093 [2] NCCL INFO Bootstrap : Using ib0:10.181.140.38<0>
ravg1038:32092:32092 [1] NCCL INFO cudaDriverVersion 12020
ravg1038:32092:32092 [1] NCCL INFO Bootstrap : Using ib0:10.181.140.38<0>
ravg1038:32094:32094 [3] NCCL INFO cudaDriverVersion 12020
ravg1038:32094:32094 [3] NCCL INFO Bootstrap : Using ib0:10.181.140.38<0>
ravg1038:32091:32091 [0] NCCL INFO cudaDriverVersion 12020
NCCL version 2.18.5+cuda12.2
ravg1038:32092:32486 [1] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
ravg1038:32092:32486 [1] NCCL INFO P2P plugin IBext
ravg1038:32092:32486 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/RoCE [RO]; OOB ib0:10.181.140.38<0>
ravg1038:32092:32486 [1] NCCL INFO Using network IBext
ravg1038:32094:32485 [3] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
ravg1038:32094:32485 [3] NCCL INFO P2P plugin IBext
ravg1038:32091:32487 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
ravg1038:32091:32487 [0] NCCL INFO P2P plugin IBext
ravg1038:32094:32485 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/RoCE [RO]; OOB ib0:10.181.140.38<0>
ravg1038:32094:32485 [3] NCCL INFO Using network IBext
ravg1038:32091:32487 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/RoCE [RO]; OOB ib0:10.181.140.38<0>
ravg1038:32091:32487 [0] NCCL INFO Using network IBext
ravg1038:32093:32484 [2] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
ravg1038:32093:32484 [2] NCCL INFO P2P plugin IBext
ravg1038:32093:32484 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/RoCE [RO]; OOB ib0:10.181.140.38<0>
ravg1038:32093:32484 [2] NCCL INFO Using network IBext
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [01:02<01:02, 62.45s/it]
Loading checkpoint shards: 50%|█████ | 1/2 [01:02<01:02, 62.36s/it]
Loading checkpoint shards: 50%|█████ | 1/2 [01:01<01:01, 61.23s/it]
Loading checkpoint shards: 50%|█████ | 1/2 [01:01<01:01, 61.12s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [01:22<00:00, 37.85s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [01:22<00:00, 41.36s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [01:22<00:00, 37.80s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [01:22<00:00, 41.30s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [01:23<00:00, 38.36s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [01:23<00:00, 41.98s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [01:23<00:00, 38.33s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [01:23<00:00, 41.94s/it]
model and tokenizer [LLAMA27BModel] were created
all params: 6,738,415,616 || trainable params: 6,738,415,616 || trainable%: 100.0
Saving the dataset (0/1 shards): 0%| | 0/8 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 968.97 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 951.84 examples/s]
Saving the dataset (0/1 shards): 0%| | 0/2 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 291.65 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 286.17 examples/s]
model and tokenizer [LLAMA27BModel] were created
all params: 6,738,415,616 || trainable params: 6,738,415,616 || trainable%: 100.0
Saving the dataset (0/1 shards): 0%| | 0/8 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 801.99 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 789.13 examples/s]
Saving the dataset (0/1 shards): 0%| | 0/2 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 159.69 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 157.81 examples/s]
model and tokenizer [LLAMA27BModel] were created
all params: 6,738,415,616 || trainable params: 6,738,415,616 || trainable%: 100.0
model and tokenizer [LLAMA27BModel] were created
all params: 6,738,415,616 || trainable params: 6,738,415,616 || trainable%: 100.0
Training has been launched...
Saving the dataset (0/1 shards): 0%| | 0/8 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 416.41 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 412.70 examples/s]
Saving the dataset (0/1 shards): 0%| | 0/2 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 130.73 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 129.33 examples/s]
ravg1040:41359:41359 [2] NCCL INFO cudaDriverVersion 12020
ravg1040:41359:41359 [2] NCCL INFO Bootstrap : Using ib0:10.181.140.40<0>
Saving the dataset (0/1 shards): 0%| | 0/8 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 599.95 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 8/8 [00:00<00:00, 592.33 examples/s]
Saving the dataset (0/1 shards): 0%| | 0/2 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 183.96 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 2/2 [00:00<00:00, 181.35 examples/s]
ravg1040:41359:41827 [2] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
ravg1040:41359:41827 [2] NCCL INFO P2P plugin IBext
ravg1040:41359:41827 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/RoCE [RO]; OOB ib0:10.181.140.40<0>
ravg1040:41359:41827 [2] NCCL INFO Using network IBext
Training has been launched...
ravg1040:41358:41358 [1] NCCL INFO cudaDriverVersion 12020
ravg1040:41358:41358 [1] NCCL INFO Bootstrap : Using ib0:10.181.140.40<0>
Training has been launched...
Training has been launched...
ravg1040:41357:41357 [0] NCCL INFO cudaDriverVersion 12020
ravg1040:41357:41357 [0] NCCL INFO Bootstrap : Using ib0:10.181.140.40<0>
ravg1040:41360:41360 [3] NCCL INFO cudaDriverVersion 12020
ravg1040:41360:41360 [3] NCCL INFO Bootstrap : Using ib0:10.181.140.40<0>
ravg1040:41358:41833 [1] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
ravg1040:41358:41833 [1] NCCL INFO P2P plugin IBext
ravg1040:41358:41833 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/RoCE [RO]; OOB ib0:10.181.140.40<0>
ravg1040:41358:41833 [1] NCCL INFO Using network IBext
ravg1040:41357:41834 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
ravg1040:41357:41834 [0] NCCL INFO P2P plugin IBext
ravg1040:41357:41834 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/RoCE [RO]; OOB ib0:10.181.140.40<0>
ravg1040:41357:41834 [0] NCCL INFO Using network IBext
ravg1040:41360:41835 [3] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so
ravg1040:41360:41835 [3] NCCL INFO P2P plugin IBext
ravg1040:41360:41835 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB/SHARP [1]mlx5_1:1/RoCE [RO]; OOB ib0:10.181.140.40<0>
ravg1040:41360:41835 [3] NCCL INFO Using network IBext
ravg1040:41360:41835 [3] NCCL INFO comm 0x5562514f5b40 rank 7 nranks 8 cudaDev 3 nvmlDev 3 busId e3000 commId 0x5683b92c7c90bf77 - Init START
ravg1038:32091:32487 [0] NCCL INFO comm 0x55f7096ee250 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 31000 commId 0x5683b92c7c90bf77 - Init START
ravg1038:32092:32486 [1] NCCL INFO comm 0x55a65588e7b0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 4b000 commId 0x5683b92c7c90bf77 - Init START
ravg1040:41359:41827 [2] NCCL INFO comm 0x558b947d9180 rank 6 nranks 8 cudaDev 2 nvmlDev 2 busId ca000 commId 0x5683b92c7c90bf77 - Init START
ravg1038:32093:32484 [2] NCCL INFO comm 0x55e645ceeb20 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId ca000 commId 0x5683b92c7c90bf77 - Init START
ravg1040:41358:41833 [1] NCCL INFO comm 0x555792dfb9c0 rank 5 nranks 8 cudaDev 1 nvmlDev 1 busId 4b000 commId 0x5683b92c7c90bf77 - Init START
ravg1038:32094:32485 [3] NCCL INFO comm 0x55ef024b7f30 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId e3000 commId 0x5683b92c7c90bf77 - Init START
ravg1040:41357:41834 [0] NCCL INFO comm 0x55ca3b72c550 rank 4 nranks 8 cudaDev 0 nvmlDev 0 busId 31000 commId 0x5683b92c7c90bf77 - Init START
ravg1040:41358:41833 [1] NCCL INFO Setting affinity for GPU 1 to 01
ravg1040:41358:41833 [1] NCCL INFO NVLS multicast support is not available on dev 1
ravg1038:32091:32487 [0] NCCL INFO Setting affinity for GPU 0 to 01
ravg1038:32091:32487 [0] NCCL INFO NVLS multicast support is not available on dev 0
ravg1038:32093:32484 [2] NCCL INFO NVLS multicast support is not available on dev 2
ravg1038:32092:32486 [1] NCCL INFO Setting affinity for GPU 1 to 01
ravg1038:32092:32486 [1] NCCL INFO NVLS multicast support is not available on dev 1
ravg1038:32094:32485 [3] NCCL INFO NVLS multicast support is not available on dev 3
ravg1040:41357:41834 [0] NCCL INFO Setting affinity for GPU 0 to 01
ravg1040:41357:41834 [0] NCCL INFO NVLS multicast support is not available on dev 0
ravg1040:41359:41827 [2] NCCL INFO NVLS multicast support is not available on dev 2
ravg1040:41360:41835 [3] NCCL INFO NVLS multicast support is not available on dev 3
ravg1040:41360:41835 [3] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6
ravg1040:41360:41835 [3] NCCL INFO P2P Chunksize set to 131072
ravg1038:32091:32487 [0] NCCL INFO Channel 00/02 : 0 3 2 1 4 7 6 5
ravg1038:32091:32487 [0] NCCL INFO Channel 01/02 : 0 3 2 1 4 7 6 5
ravg1038:32091:32487 [0] NCCL INFO Trees [0] 1/4/-1->0->-1 [1] 1/-1/-1->0->4
ravg1038:32091:32487 [0] NCCL INFO P2P Chunksize set to 131072
ravg1038:32092:32486 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0
ravg1038:32092:32486 [1] NCCL INFO P2P Chunksize set to 131072
ravg1038:32094:32485 [3] NCCL INFO Trees [0] -1/-1/-1->3->2 [1] -1/-1/-1->3->2
ravg1038:32094:32485 [3] NCCL INFO P2P Chunksize set to 131072
ravg1038:32093:32484 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1
ravg1038:32093:32484 [2] NCCL INFO P2P Chunksize set to 131072
ravg1040:41358:41833 [1] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4
ravg1040:41357:41834 [0] NCCL INFO Trees [0] 5/-1/-1->4->0 [1] 5/0/-1->4->-1
ravg1040:41357:41834 [0] NCCL INFO P2P Chunksize set to 131072
ravg1040:41359:41827 [2] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5
ravg1040:41359:41827 [2] NCCL INFO P2P Chunksize set to 131072
ravg1040:41358:41833 [1] NCCL INFO P2P Chunksize set to 131072
ravg1038:32091:32487 [0] NCCL INFO Channel 00/0 : 5[1] -> 0[0] [receive] via NET/IBext/0
ravg1038:32093:32484 [2] NCCL INFO Channel 00/0 : 2[2] -> 1[1] via P2P/IPC/read
ravg1040:41358:41833 [1] NCCL INFO Channel 00/0 : 5[1] -> 0[0] [send] via NET/IBext/0
ravg1038:32091:32487 [0] NCCL INFO Channel 01/0 : 5[1] -> 0[0] [receive] via NET/IBext/0
ravg1038:32091:32487 [0] NCCL INFO Channel 00/0 : 0[0] -> 3[3] via P2P/IPC/read
ravg1038:32092:32486 [1] NCCL INFO Channel 00/0 : 1[1] -> 4[0] [send] via NET/IBext/0
ravg1040:41359:41827 [2] NCCL INFO Channel 00/0 : 6[2] -> 5[1] via P2P/IPC/read
ravg1040:41359:41827 [2] NCCL INFO Channel 01/0 : 6[2] -> 5[1] via P2P/IPC/read
ravg1040:41357:41834 [0] NCCL INFO Channel 00/0 : 1[1] -> 4[0] [receive] via NET/IBext/0
ravg1040:41357:41834 [0] NCCL INFO Channel 01/0 : 1[1] -> 4[0] [receive] via NET/IBext/0
ravg1040:41357:41834 [0] NCCL INFO Channel 00/0 : 4[0] -> 7[3] via P2P/IPC/read
ravg1040:41357:41834 [0] NCCL INFO Channel 01/0 : 4[0] -> 7[3] via P2P/IPC/read
ravg1040:41358:41833 [1] NCCL INFO Channel 01/0 : 5[1] -> 0[0] [send] via NET/IBext/0
ravg1040:41360:41835 [3] NCCL INFO Channel 00/0 : 7[3] -> 6[2] via P2P/IPC/read
ravg1040:41360:41835 [3] NCCL INFO Channel 01/0 : 7[3] -> 6[2] via P2P/IPC/read
ravg1038:32093:32484 [2] NCCL INFO Channel 01/0 : 2[2] -> 1[1] via P2P/IPC/read
ravg1038:32091:32487 [0] NCCL INFO Channel 01/0 : 0[0] -> 3[3] via P2P/IPC/read
ravg1040:41360:41835 [3] NCCL INFO Connected all rings
ravg1038:32092:32486 [1] NCCL INFO Channel 01/0 : 1[1] -> 4[0] [send] via NET/IBext/0
ravg1038:32094:32485 [3] NCCL INFO Channel 00/0 : 3[3] -> 2[2] via P2P/IPC/read
ravg1038:32094:32485 [3] NCCL INFO Channel 01/0 : 3[3] -> 2[2] via P2P/IPC/read
ravg1038:32094:32485 [3] NCCL INFO Connected all rings
ravg1040:41359:41827 [2] NCCL INFO Connected all rings
ravg1038:32093:32484 [2] NCCL INFO Connected all rings
ravg1040:41358:41833 [1] NCCL INFO Connected all rings
ravg1040:41359:41827 [2] NCCL INFO Channel 00/0 : 6[2] -> 7[3] via P2P/IPC/read
ravg1040:41359:41827 [2] NCCL INFO Channel 01/0 : 6[2] -> 7[3] via P2P/IPC/read
ravg1040:41360:41835 [3] NCCL INFO Connected all trees
ravg1040:41360:41835 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
ravg1040:41360:41835 [3] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
ravg1040:41358:41833 [1] NCCL INFO Channel 00/0 : 5[1] -> 6[2] via P2P/IPC/read
ravg1038:32092:32486 [1] NCCL INFO Connected all rings
ravg1038:32092:32486 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/IPC/read
ravg1038:32092:32486 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/IPC/read
ravg1038:32091:32487 [0] NCCL INFO Connected all rings
ravg1038:32091:32487 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/IPC/read
ravg1038:32093:32484 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/IPC/read
ravg1038:32091:32487 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/IPC/read
ravg1038:32091:32487 [0] NCCL INFO Channel 00/0 : 4[0] -> 0[0] [receive] via NET/IBext/0
ravg1038:32091:32487 [0] NCCL INFO Channel 01/0 : 4[0] -> 0[0] [receive] via NET/IBext/0
ravg1038:32091:32487 [0] NCCL INFO Channel 00/0 : 0[0] -> 4[0] [send] via NET/IBext/0
ravg1038:32091:32487 [0] NCCL INFO Channel 01/0 : 0[0] -> 4[0] [send] via NET/IBext/0
ravg1038:32093:32484 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/IPC/read
ravg1038:32092:32486 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/IPC/read
ravg1040:41357:41834 [0] NCCL INFO Connected all rings
ravg1040:41357:41834 [0] NCCL INFO Channel 00/0 : 4[0] -> 5[1] via P2P/IPC/read
ravg1040:41357:41834 [0] NCCL INFO Channel 01/0 : 4[0] -> 5[1] via P2P/IPC/read
ravg1038:32094:32485 [3] NCCL INFO Connected all trees
ravg1038:32094:32485 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
ravg1038:32094:32485 [3] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
ravg1040:41358:41833 [1] NCCL INFO Channel 01/0 : 5[1] -> 6[2] via P2P/IPC/read
ravg1038:32092:32486 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/IPC/read
ravg1040:41358:41833 [1] NCCL INFO Channel 00/0 : 5[1] -> 4[0] via P2P/IPC/read
ravg1040:41357:41834 [0] NCCL INFO Channel 00/0 : 0[0] -> 4[0] [receive] via NET/IBext/0
ravg1040:41357:41834 [0] NCCL INFO Channel 01/0 : 0[0] -> 4[0] [receive] via NET/IBext/0
ravg1040:41358:41833 [1] NCCL INFO Channel 01/0 : 5[1] -> 4[0] via P2P/IPC/read
ravg1040:41357:41834 [0] NCCL INFO Channel 00/0 : 4[0] -> 0[0] [send] via NET/IBext/0
ravg1038:32093:32484 [2] NCCL INFO Connected all trees
ravg1038:32093:32484 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
ravg1038:32093:32484 [2] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
ravg1040:41357:41834 [0] NCCL INFO Channel 01/0 : 4[0] -> 0[0] [send] via NET/IBext/0
ravg1040:41359:41827 [2] NCCL INFO Connected all trees
ravg1040:41359:41827 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
ravg1040:41359:41827 [2] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
ravg1040:41358:41833 [1] NCCL INFO Connected all trees
ravg1040:41358:41833 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
ravg1040:41358:41833 [1] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
ravg1038:32092:32486 [1] NCCL INFO Connected all trees
ravg1038:32092:32486 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
ravg1038:32092:32486 [1] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
ravg1040:41357:41834 [0] NCCL INFO Connected all trees
ravg1040:41357:41834 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
ravg1040:41357:41834 [0] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
ravg1038:32091:32487 [0] NCCL INFO Connected all trees
ravg1038:32091:32487 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512
ravg1038:32091:32487 [0] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
ravg1040:41358:41833 [1] NCCL INFO comm 0x555792dfb9c0 rank 5 nranks 8 cudaDev 1 nvmlDev 1 busId 4b000 commId 0x5683b92c7c90bf77 - Init COMPLETE
ravg1040:41360:41835 [3] NCCL INFO comm 0x5562514f5b40 rank 7 nranks 8 cudaDev 3 nvmlDev 3 busId e3000 commId 0x5683b92c7c90bf77 - Init COMPLETE
ravg1040:41359:41827 [2] NCCL INFO comm 0x558b947d9180 rank 6 nranks 8 cudaDev 2 nvmlDev 2 busId ca000 commId 0x5683b92c7c90bf77 - Init COMPLETE
ravg1040:41357:41834 [0] NCCL INFO comm 0x55ca3b72c550 rank 4 nranks 8 cudaDev 0 nvmlDev 0 busId 31000 commId 0x5683b92c7c90bf77 - Init COMPLETE
ravg1038:32092:32486 [1] NCCL INFO comm 0x55a65588e7b0 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 4b000 commId 0x5683b92c7c90bf77 - Init COMPLETE
ravg1038:32093:32484 [2] NCCL INFO comm 0x55e645ceeb20 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId ca000 commId 0x5683b92c7c90bf77 - Init COMPLETE
ravg1038:32094:32485 [3] NCCL INFO comm 0x55ef024b7f30 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId e3000 commId 0x5683b92c7c90bf77 - Init COMPLETE
ravg1038:32091:32487 [0] NCCL INFO comm 0x55f7096ee250 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 31000 commId 0x5683b92c7c90bf77 - Init COMPLETE
Traceback (most recent call last):
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 125, in
train(opt)
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 118, in train
trainer.train()
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1555, in train
return inner_training_loop(
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1689, in _inner_training_loop
model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1288, in prepare
result = tuple(
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1289, in
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1094, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1433, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 811, in init
self._ddp_init_helper(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 1100, in _ddp_init_helper
self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 25.10 GiB. GPU 0 has a total capacty of 39.39 GiB of which 13.58 GiB is free. Including non-PyTorch memory, this process has 25.80 GiB memory in use. Of the allocated memory 25.23 GiB is allocated by PyTorch, and 12.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 125, in
train(opt)
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 118, in train
trainer.train()
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1555, in train
return inner_training_loop(
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1689, in _inner_training_loop
model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1288, in prepare
result = tuple(
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1289, in
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1094, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1433, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 811, in init
self._ddp_init_helper(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 1100, in _ddp_init_helper
self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 25.10 GiB. GPU 2 has a total capacty of 39.39 GiB of which 13.57 GiB is free. Including non-PyTorch memory, this process has 25.81 GiB memory in use. Of the allocated memory 25.23 GiB is allocated by PyTorch, and 12.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 125, in
train(opt)
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 118, in train
trainer.train()
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1555, in train
return inner_training_loop(
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1689, in _inner_training_loop
model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1288, in prepare
result = tuple(
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1289, in
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1094, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1433, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 811, in init
self._ddp_init_helper(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 1100, in _ddp_init_helper
self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 25.10 GiB. GPU 1 has a total capacty of 39.39 GiB of which 13.57 GiB is free. Including non-PyTorch memory, this process has 25.81 GiB memory in use. Of the allocated memory 25.23 GiB is allocated by PyTorch, and 12.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 125, in
train(opt)
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 118, in train
trainer.train()
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1555, in train
return inner_training_loop(
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1689, in _inner_training_loop
model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1288, in prepare
result = tuple(
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1289, in
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1094, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1433, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 811, in init
self._ddp_init_helper(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 1100, in _ddp_init_helper
self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 25.10 GiB. GPU 3 has a total capacty of 39.39 GiB of which 13.58 GiB is free. Including non-PyTorch memory, this process has 25.80 GiB memory in use. Of the allocated memory 25.23 GiB is allocated by PyTorch, and 12.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 125, in
train(opt)
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 118, in train
trainer.train()
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1555, in train
return inner_training_loop(
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1689, in _inner_training_loop
model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1288, in prepare
result = tuple(
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1289, in
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1094, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1433, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 811, in init
self._ddp_init_helper(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 1100, in _ddp_init_helper
self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 25.10 GiB. GPU 2 has a total capacty of 39.39 GiB of which 13.57 GiB is free. Including non-PyTorch memory, this process has 25.81 GiB memory in use. Of the allocated memory 25.23 GiB is allocated by PyTorch, and 12.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 125, in
train(opt)
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 118, in train
trainer.train()
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1555, in train
return inner_training_loop(
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1689, in _inner_training_loop
model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1288, in prepare
result = tuple(
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1289, in
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1094, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1433, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 811, in init
self._ddp_init_helper(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 1100, in _ddp_init_helper
self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 25.10 GiB. GPU 0 has a total capacty of 39.39 GiB of which 13.58 GiB is free. Including non-PyTorch memory, this process has 25.80 GiB memory in use. Of the allocated memory 25.23 GiB is allocated by PyTorch, and 12.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 125, in
train(opt)
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 118, in train
trainer.train()
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1555, in train
return inner_training_loop(
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1689, in _inner_training_loop
model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1288, in prepare
result = tuple(
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1289, in
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1094, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1433, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 811, in init
self._ddp_init_helper(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 1100, in _ddp_init_helper
self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 25.10 GiB. GPU 3 has a total capacty of 39.39 GiB of which 13.58 GiB is free. Including non-PyTorch memory, this process has 25.80 GiB memory in use. Of the allocated memory 25.23 GiB is allocated by PyTorch, and 12.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 125, in
train(opt)
File "/llm_fine_tuning/src/fine_tuning_llm.py", line 118, in train
trainer.train()
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1555, in train
return inner_training_loop(
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1689, in _inner_training_loop
model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1288, in prepare
result = tuple(
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1289, in
self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1094, in _prepare_one
return self.prepare_model(obj, device_placement=device_placement)
File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1433, in prepare_model
model = torch.nn.parallel.DistributedDataParallel(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 811, in init
self._ddp_init_helper(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py", line 1100, in _ddp_init_helper
self.reducer = dist.Reducer(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 25.10 GiB. GPU 1 has a total capacty of 39.39 GiB of which 13.57 GiB is free. Including non-PyTorch memory, this process has 25.81 GiB memory in use. Of the allocated memory 25.23 GiB is allocated by PyTorch, and 12.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2023-11-22 11:10:17,084] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 41357 closing signal SIGTERM
[2023-11-22 11:10:17,084] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 41359 closing signal SIGTERM
[2023-11-22 11:10:17,084] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 41360 closing signal SIGTERM
[2023-11-22 11:10:17,267] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 32091) of binary: /usr/bin/python
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
/llm_fine_tuning/src/fine_tuning_llm.py FAILED
Failures:
[1]:
time : 2023-11-22_11:10:17
host : ravg1038.hpc.mpcdf.mpg.de
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 32092)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2023-11-22_11:10:17
host : ravg1038.hpc.mpcdf.mpg.de
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 32093)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
time : 2023-11-22_11:10:17
host : ravg1038.hpc.mpcdf.mpg.de
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 32094)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
Root Cause (first observed failure):
[0]:
time : 2023-11-22_11:10:17
host : ravg1038.hpc.mpcdf.mpg.de
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 32091)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2023-11-22 11:10:17,699] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 41358) of binary: /usr/bin/python
[2023-11-22 11:10:17,708] torch.distributed.elastic.rendezvous.dynamic_rendezvous: [WARNING] The node 'ravg1040.hpc.mpcdf.mpg.de_41263_0' has failed to shutdown the rendezvous '7957518' due to an error of type RendezvousConnectionError.
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
System info (please complete the following information):
- OS:
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy - GPU: two machines with x4 A100s each
- Python 3.10.12
Launcher context
#!/bin/bash -l
#SBATCH -J transformers
#SBATCH --nodes=2
#SBATCH --tasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=0
#SBATCH --constraint="gpu"
#SBATCH --gres=gpu:a100:4
Wall clock limit (max. is 24 hours):
#SBATCH --time=00:15:00
#SBATCH --output=slurm_logs/tests/slurm-%x-%j.out
source /etc/profile.d/modules.sh
module purge
module load apptainer
module load git-lfs/3.3
git lfs install
git lfs pull
Avoid hyper-threading (in this case cpus-per-task // nr_of_gpus):
export OMP_NUM_THREADS=18
For pinning threads correctly:
export OMP_PLACES=cores
Useful for debugging:
export NCCL_DEBUG=INFO
srun apptainer exec
--nv -B .:/llm_fine_tuning
images/deepspeed_torchrun_00.sif torchrun
--nnodes="$SLURM_NNODES"
--nproc-per-node=gpu
--rdzv-id="$SLURM_JOBID"
--rdzv-endpoint=$(scontrol show hostnames "$SLURM_JOB_NODELIST" | head -n 1)
--rdzv-backend="c10d"
/llm_fine_tuning/src/fine_tuning_llm.py --config_file '/llm_fine_tuning/src/tests/cluster/config/full.yml'
Docker context
BootStrap: docker
From: nvcr.io/nvidia/pytorch:23.09-py3
%post
mkdir /llm_fine_tuning
apt-get update &&
apt-get install -y --no-install-recommends
libaio-dev
DS_BUILD_OPS=1 DS_BUILD_SPARSE_ATTN=0 DS_BUILD_EVOFORMER_ATTN=0 DS_BUILD_CUTLASS_OPS=0 DS_BUILD_RAGGED_DEVICE_OPS=0 pip install deepspeed
cd installation
pip install --upgrade pip && pip --no-cache-dir install -e ".[full]" -i https://pypi.org/simple
pip install accelerate datasets transformers peft
%environment
%files
src /installation/src
setup.py /installation/setup.py
requirements.txt /installation/requirements.txt
requirements_dev.txt /installation/requirements_dev.txt
requirements_cluster.txt /installation/requirements_cluster.txt
Additional context
ds_config:
{
"optimizer": {
"type": "AdamW",
"params": {
"lr": "auto",
"weight_decay": "auto"
}
},
"scheduler": {
"type": "WarmupDecayLR",
"params": {
"warmup_min_lr": "auto",
"warmup_max_lr": "auto",
"warmup_num_steps": "auto",
"total_num_steps": "auto"
}
},
"zero_optimization": {
"stage": 3,
"offload_optimizer": {
"device": "cpu",
"pin_memory": true
},
"offload_param": {
"device": "cpu",
"pin_memory": true
},
"overlap_comm": true,
"contiguous_gradients": true,
"reduce_bucket_size": "auto",
"stage3_prefetch_bucket_size": "auto",
"stage3_param_persistence_threshold": "auto",
"sub_group_size": 1e9,
"stage3_max_live_parameters": 1e9,
"stage3_max_reuse_distance": 1e9,
"stage3_gather_16bit_weights_on_model_save": "auto"
},
"gradient_accumulation_steps": "auto",
"gradient_clipping": "auto",
"steps_per_print": 2000,
"train_batch_size": "auto",
"train_micro_batch_size_per_gpu": "auto",
"wall_clock_breakdown": false,
"wandb": {
"enabled": true
}
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the cluster configuration in /llm_fine_tuning/src/tests/cluster/config/full.yml and the DeepSpeed settings in /llm_fine_tuning/src/ds_config.json, then compare them with the Torchrun setup and ds_report output. Reproduce the multi-node Llama 2 7B training run and identify the training-stage CUDA out-of-memory failure. Done means the reported configuration trains without that OOM or the issue has a documented, reproducible cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100