deepspeedai / deepspeedai/DeepSpeed

[BUG] deepspeed inferene opt/66b model OOM on 8GPUs

Open
#2,933 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug inference
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

My deepspeed is 0.8.1 and transformers is 4.21.2 and I have 8 V100 32GB on my machine

  • I try the code ds_inference and set the mp_size to be 2/4/8.
  • The below is my error logs for mp_size = 4/8.
03-03 03:41:09,338] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed info: version=0.8.1, git-hash=unknown, git-branch=unknown
[2023-03-03 03:41:09,339] [WARNING] [config_utils.py:74:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-03-03 03:41:09,339] [WARNING] [config_utils.py:74:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-03-03 03:41:09,339] [WARNING] [config_utils.py:74:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-03-03 03:41:09,339] [WARNING] [config_utils.py:74:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-03-03 03:41:09,339] [WARNING] [config_utils.py:74:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-03-03 03:41:09,339] [WARNING] [config_utils.py:74:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-03-03 03:41:09,339] [WARNING] [config_utils.py:74:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-03-03 03:41:09,340] [WARNING] [config_utils.py:74:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead
[2023-03-03 03:41:09,340] [INFO] [logging.py:75:log_dist] [Rank 0] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /home/lambda7xx/.cache/torch_extensions/py38_cu117/transformer_inference/build.ninja...
Building extension module transformer_inference...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.5223760604858398 seconds
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.5085861682891846 seconds
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.5094120502471924 seconds
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.5131995677947998 seconds
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.5160329341888428 seconds
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.5114881992340088 seconds
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.5109550952911377 seconds
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.6042768955230713 seconds
[2023-03-03 03:41:10,919] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed-Inference config: {'layer_id': 0, 'hidden_size': 9216, 'intermediate_size': 36864, 'heads': 72, 'num_hidden_layers': -1, 'fp16': True, 'pre_layer_norm': True, 'local_rank': -1, 'stochastic_mode': False, 'epsilon': 1e-12, 'mp_size': 4, 'q_int8': False, 'scale_attention': True, 'triangular_masking': True, 'local_attention': False, 'window_size': 1, 'rotary_dim': -1, 'rotate_half': False, 'rotate_every_two': True, 'return_tuple': True, 'mlp_after_attn': True, 'mlp_act_func_type': <ActivationFuncType.ReLU: 2>, 'specialized_mode': False, 'training_mp_size': 1, 'bigscience_bloom': False, 'max_out_tokens': 1024, 'scale_attn_by_inverse_layer_idx': False, 'enable_qkv_quantization': False, 'use_mup': False, 'return_single_tuple': False}
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Installed CUDA version 11.8 does not match the version torch was compiled with 11.7 but since the APIs are compatible, accepting this combination
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.11136794090270996 seconds
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.10508179664611816 seconds
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.10427236557006836 seconds
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.1050100326538086 seconds
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.10665178298950195 seconds
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.10624575614929199 seconds
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.1120920181274414 seconds
Using /home/lambda7xx/.cache/torch_extensions/py38_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module transformer_inference, skipping build step...
Loading extension module transformer_inference...
Time to load transformer_inference op: 0.10815262794494629 seconds

Loading 14 checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s]
Loading 14 checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s]
Loading 14 checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s]
Loading 14 checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s]
Loading 14 checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s]
Loading 14 checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s]
Loading 14 checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s]
Loading 14 checkpoint shards:   0%|          | 0/14 [00:00<?, ?it/s]Traceback (most recent call last):
  File "bloom-inference-scripts/bloom-ds-inference.py", line 185, in <module>
    model = deepspeed.init_inference(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/__init__.py", line 311, in init_inference
    engine = InferenceEngine(model, config=ds_inference_config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 134, in __init__
    self._apply_injection_policy(config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 358, in _apply_injection_policy
    replace_transformer_layer(client_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 561, in replace_transformer_layer
    load_model_with_checkpoint(replaced_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 279, in load_model_with_checkpoint
    load_module_recursive(r_module)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 271, in load_module_recursive
    layer_policies[child.__class__](child, prefix + name + '.')
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 200, in load_transformer_layer
    replace_policy.load_params(module,
AttributeError: 'HFOPTLayerPolicy' object has no attribute 'load_params'
Traceback (most recent call last):
  File "bloom-inference-scripts/bloom-ds-inference.py", line 185, in <module>
    model = deepspeed.init_inference(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/__init__.py", line 311, in init_inference
    engine = InferenceEngine(model, config=ds_inference_config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 134, in __init__
    self._apply_injection_policy(config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 358, in _apply_injection_policy
    replace_transformer_layer(client_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 561, in replace_transformer_layer
    load_model_with_checkpoint(replaced_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 279, in load_model_with_checkpoint
    load_module_recursive(r_module)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 271, in load_module_recursive
    layer_policies[child.__class__](child, prefix + name + '.')
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 200, in load_transformer_layer
    replace_policy.load_params(module,
AttributeError: 'HFOPTLayerPolicy' object has no attribute 'load_params'

Loading 14 checkpoint shards:   0%|          | 0/14 [00:05<?, ?it/s]
Traceback (most recent call last):
  File "bloom-inference-scripts/bloom-ds-inference.py", line 185, in <module>
    model = deepspeed.init_inference(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/__init__.py", line 311, in init_inference
    engine = InferenceEngine(model, config=ds_inference_config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 134, in __init__
    self._apply_injection_policy(config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 358, in _apply_injection_policy
    replace_transformer_layer(client_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 561, in replace_transformer_layer
    load_model_with_checkpoint(replaced_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 279, in load_model_with_checkpoint
    load_module_recursive(r_module)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 271, in load_module_recursive
    layer_policies[child.__class__](child, prefix + name + '.')
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 200, in load_transformer_layer
    replace_policy.load_params(module,
AttributeError: 'HFOPTLayerPolicy' object has no attribute 'load_params'
Traceback (most recent call last):
  File "bloom-inference-scripts/bloom-ds-inference.py", line 185, in <module>
    model = deepspeed.init_inference(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/__init__.py", line 311, in init_inference
    engine = InferenceEngine(model, config=ds_inference_config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 134, in __init__
    self._apply_injection_policy(config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 358, in _apply_injection_policy
    replace_transformer_layer(client_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 561, in replace_transformer_layer
    load_model_with_checkpoint(replaced_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 279, in load_model_with_checkpoint
    load_module_recursive(r_module)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 271, in load_module_recursive
    layer_policies[child.__class__](child, prefix + name + '.')
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 200, in load_transformer_layer
    replace_policy.load_params(module,
AttributeError: 'HFOPTLayerPolicy' object has no attribute 'load_params'

Loading 14 checkpoint shards:   0%|          | 0/14 [00:05<?, ?it/s]

Loading 14 checkpoint shards:   0%|          | 0/14 [00:05<?, ?it/s]

Loading 14 checkpoint shards:   0%|          | 0/14 [00:05<?, ?it/s]
Traceback (most recent call last):
  File "bloom-inference-scripts/bloom-ds-inference.py", line 185, in <module>
    model = deepspeed.init_inference(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/__init__.py", line 311, in init_inference
    engine = InferenceEngine(model, config=ds_inference_config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 134, in __init__
    self._apply_injection_policy(config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 358, in _apply_injection_policy
    replace_transformer_layer(client_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 561, in replace_transformer_layer
    load_model_with_checkpoint(replaced_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 279, in load_model_with_checkpoint
    load_module_recursive(r_module)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 271, in load_module_recursive
    layer_policies[child.__class__](child, prefix + name + '.')
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 200, in load_transformer_layer
    replace_policy.load_params(module,
AttributeError: 'HFOPTLayerPolicy' object has no attribute 'load_params'
Traceback (most recent call last):
  File "bloom-inference-scripts/bloom-ds-inference.py", line 185, in <module>
    model = deepspeed.init_inference(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/__init__.py", line 311, in init_inference
    engine = InferenceEngine(model, config=ds_inference_config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 134, in __init__
    self._apply_injection_policy(config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 358, in _apply_injection_policy
    replace_transformer_layer(client_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 561, in replace_transformer_layer
    load_model_with_checkpoint(replaced_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 279, in load_model_with_checkpoint
    load_module_recursive(r_module)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 271, in load_module_recursive
    layer_policies[child.__class__](child, prefix + name + '.')
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 200, in load_transformer_layer
    replace_policy.load_params(module,
AttributeError: 'HFOPTLayerPolicy' object has no attribute 'load_params'

Loading 14 checkpoint shards:   0%|          | 0/14 [00:06<?, ?it/s]

Loading 14 checkpoint shards:   0%|          | 0/14 [00:06<?, ?it/s]
PHLRR4036:4558:5025 [4] NCCL INFO [Service thread] Connection closed by localRank 4
Traceback (most recent call last):
  File "bloom-inference-scripts/bloom-ds-inference.py", line 185, in <module>
    model = deepspeed.init_inference(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/__init__.py", line 311, in init_inference
    engine = InferenceEngine(model, config=ds_inference_config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 134, in __init__
    self._apply_injection_policy(config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 358, in _apply_injection_policy
    replace_transformer_layer(client_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 561, in replace_transformer_layer
    load_model_with_checkpoint(replaced_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 279, in load_model_with_checkpoint
    load_module_recursive(r_module)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 271, in load_module_recursive
    layer_policies[child.__class__](child, prefix + name + '.')
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 200, in load_transformer_layer
    replace_policy.load_params(module,
AttributeError: 'HFOPTLayerPolicy' object has no attribute 'load_params'
PHLRR4036:4558:4558 [4] NCCL INFO comm 0x4a158970 rank 4 nranks 8 cudaDev 4 busId 83000 - Abort COMPLETE
Traceback (most recent call last):
  File "bloom-inference-scripts/bloom-ds-inference.py", line 185, in <module>
    model = deepspeed.init_inference(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/__init__.py", line 311, in init_inference
    engine = InferenceEngine(model, config=ds_inference_config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 134, in __init__
    self._apply_injection_policy(config)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/inference/engine.py", line 358, in _apply_injection_policy
    replace_transformer_layer(client_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/replace_module.py", line 561, in replace_transformer_layer
    load_model_with_checkpoint(replaced_module,
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 279, in load_model_with_checkpoint
    load_module_recursive(r_module)
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 273, in load_module_recursive
    load_module_recursive(
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 271, in load_module_recursive
    layer_policies[child.__class__](child, prefix + name + '.')
  File "/home/lambda7xx/.local/lib/python3.8/site-packages/deepspeed/module_inject/load_checkpoint.py", line 200, in load_transformer_layer
    replace_policy.load_params(module,
AttributeError: 'HFOPTLayerPolicy' object has no attribute 'load_params'
PHLRR4036:4556:5028 [3] NCCL INFO [Service thread] Connection closed by localRank 3
PHLRR4036:4556:4556 [3] NCCL INFO comm 0x4a9c1610 rank 3 nranks 8 cudaDev 3 busId 13000 - Abort COMPLETE

Loading 14 checkpoint shards:   0%|          | 0/14 [00:06<?, ?it/s]
PHLRR4036:4560:5029 [5] NCCL INFO [Service thread] Connection closed by localRank 5
PHLRR4036:4560:4560 [5] NCCL INFO comm 0x498b20b0 rank 5 nranks 8 cudaDev 5 busId 89000 - Abort COMPLETE

Loading 14 checkpoint shards:   0%|          | 0/14 [00:06<?, ?it/s]
PHLRR4036:4553:5017 [0] NCCL INFO [Service thread] Connection closed by localRank 0
PHLRR4036:4553:4553 [0] NCCL INFO comm 0x4b092180 rank 0 nranks 8 cudaDev 0 busId 5000 - Abort COMPLETE
PHLRR4036:4554:5024 [1] NCCL INFO [Service thread] Connection closed by localRank 1
PHLRR4036:4554:4554 [1] NCCL INFO comm 0x4a1b1a70 rank 1 nranks 8 cudaDev 1 busId 8000 - Abort COMPLETE
PHLRR4036:4564:5026 [7] NCCL INFO [Service thread] Connection closed by localRank 7
PHLRR4036:4564:4564 [7] NCCL INFO comm 0x4a6a3370 rank 7 nranks 8 cudaDev 7 busId 91000 - Abort COMPLETE
PHLRR4036:4562:5027 [6] NCCL INFO [Service thread] Connection closed by localRank 6
PHLRR4036:4555:5023 [2] NCCL INFO [Service thread] Connection closed by localRank 2
PHLRR4036:4562:4562 [6] NCCL INFO comm 0x4944deb0 rank 6 nranks 8 cudaDev 6 busId 8e000 - Abort COMPLETE
PHLRR4036:4555:4555 [2] NCCL INFO comm 0x47643180 rank 2 nranks 8 cudaDev 2 busId d000 - Abort COMPLETE
[2023-03-03 03:41:22,399] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 4553
[2023-03-03 03:41:22,433] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 4554
[2023-03-03 03:41:23,542] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 4555
[2023-03-03 03:41:23,839] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 4556
[2023-03-03 03:41:23,842] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 4558
[2023-03-03 03:41:23,843] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 4560
[2023-03-03 03:41:23,846] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 4562
[2023-03-03 03:41:23,848] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 4564
[2023-03-03 03:41:23,850] [ERROR] [launch.py:324:sigkill_handler] ['/usr/bin/python3', '-u', 'bloom-inference-scripts/bloom-ds-inference.py', '--local_rank=7', '--name', 'facebook/opt-66b', '--batch_size', '4', '--tp_size', '4', '--benchmark'] exits with return code = 1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with bloom-inference-scripts/bloom-ds-inference.py around line 185, where deepspeed.init_inference is called, and follow the reported traceback through DeepSpeed's inference initialization and checkpoint loading. Reproduce the OOM with the stated DeepSpeed, Transformers, CUDA, GPU, and mp_size settings, then document or verify the cause and expected inference configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.