deepspeedai / deepspeedai/DeepSpeed
[BUG] Errors when running with Microsoft Phi models
Open
Nobody has claimed this yet.
bug
inference
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
Tensor parallelism doesn't work with Microsoft Phi models.
To Reproduce
import torch
from deepspeed import init_inference
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2")
ds_engine = init_inference(model, tensor_parallel={"tp_size": 2})
input_ids = torch.randint(0, 100, (2, 10), dtype=torch.long).to("cuda")
ds_engine.generate(input_ids)
Expected behavior
Raises an error:
[2024-02-27 08:11:15,763] torch.distributed.run: [WARNING]
[2024-02-27 08:11:15,763] torch.distributed.run: [WARNING] *****************************************
[2024-02-27 08:11:15,763] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2024-02-27 08:11:15,763] torch.distributed.run: [WARNING] *****************************************
[2024-02-27 08:11:17,549] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-02-27 08:11:17,562] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 863/863 [00:00<00:00, 5.30MB/s]
model.safetensors.index.json: 100%|███████████████████████████████████████████████████████████████████████████| 35.7k/35.7k [00:00<00:00, 6.05MB/s]
model-00001-of-00002.safetensors: 100%|███████████████████████████████████████████████████████████████████████| 5.00G/5.00G [01:40<00:00, 49.8MB/s]
model-00002-of-00002.safetensors: 100%|█████████████████████████████████████████████████████████████████████████| 564M/564M [00:09<00:00, 58.8MB/s]
Downloading shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:50<00:00, 55.37s/it]
Downloading shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:50<00:00, 55.42s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00, 1.72s/it]
generation_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████| 124/124 [00:00<00:00, 715kB/s]
[2024-02-27 08:13:13,705] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.13.4, git-hash=unknown, git-branch=unknown
[2024-02-27 08:13:13,707] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:03<00:00, 1.86s/it]
[2024-02-27 08:13:13,865] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.13.4, git-hash=unknown, git-branch=unknown
[2024-02-27 08:13:13,867] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
[2024-02-27 08:13:16,100] [INFO] [comm.py:637:init_distributed] cdb=None
[2024-02-27 08:13:16,100] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
AutoTP: [(<class 'transformers.models.phi.modeling_phi.PhiDecoderLayer'>, ['mlp.fc2'])]
[2024-02-27 08:13:16,250] [INFO] [comm.py:637:init_distributed] cdb=None
AutoTP: [(<class 'transformers.models.phi.modeling_phi.PhiDecoderLayer'>, ['mlp.fc2'])]
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
/home/user/.local/lib/python3.10/site-packages/transformers/generation/utils.py:1178: UserWarning: Using the model-agnostic default `max_length` (=20) to control the generation length. We recommend setting `max_new_tokens` to control the maximum length of the generation.
warnings.warn(
/home/user/.local/lib/python3.10/site-packages/transformers/generation/utils.py:1178: UserWarning: Using the model-agnostic default `max_length` (=20) to control the generation length. We recommend setting `max_new_tokens` to control the maximum length of the generation.
warnings.warn(
Traceback (most recent call last):
File "/workspace/minimal.py", line 9, in <module>
ds_engine.generate(input_ids)
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/inference/engine.py", line 636, in _generate
return self.module.generate(*inputs, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/transformers/generation/utils.py", line 1544, in generate
return self.greedy_search(
File "/home/user/.local/lib/python3.10/site-packages/transformers/generation/utils.py", line 2404, in greedy_search
outputs = self(
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/transformers/models/phi/modeling_phi.py", line 1046, in forward
outputs = self.model(
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/transformers/models/phi/modeling_phi.py", line 925, in forward
layer_outputs = decoder_layer(
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
Traceback (most recent call last):
File "/workspace/minimal.py", line 9, in <module>
return self._call_impl(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
ds_engine.generate(input_ids)
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/inference/engine.py", line 636, in _generate
return forward_call(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/transformers/models/phi/modeling_phi.py", line 666, in forward
return self.module.generate(*inputs, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
attn_outputs, self_attn_weights, present_key_value = self.self_attn( File "/home/user/.local/lib/python3.10/site-packages/transformers/generation/utils.py", line 1544, in generate
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return self.greedy_search(
File "/home/user/.local/lib/python3.10/site-packages/transformers/generation/utils.py", line 2404, in greedy_search
return forward_call(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/transformers/models/phi/modeling_phi.py", line 395, in forward
attn_output = self.dense(attn_output)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
outputs = self(
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/module_inject/layers.py", line 80, in forward
output = torch.matmul(input, self.weight.transpose(-1, -2))
RuntimeError: mat1 and mat2 shapes cannot be multiplied (20x1280 and 2560x1280)
return forward_call(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/transformers/models/phi/modeling_phi.py", line 1046, in forward
outputs = self.model(
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/transformers/models/phi/modeling_phi.py", line 925, in forward
layer_outputs = decoder_layer(
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/transformers/models/phi/modeling_phi.py", line 666, in forward
attn_outputs, self_attn_weights, present_key_value = self.self_attn(
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/transformers/models/phi/modeling_phi.py", line 395, in forward
attn_output = self.dense(attn_output)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/deepspeed/module_inject/layers.py", line 80, in forward
output = torch.matmul(input, self.weight.transpose(-1, -2))
RuntimeError: mat1 and mat2 shapes cannot be multiplied (20x1280 and 2560x1280)
[2024-02-27 08:13:20,897] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 360939) of binary: /usr/bin/python3
Traceback (most recent call last):
File "/home/user/.local/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/home/user/.local/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
File "/home/user/.local/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
run(args)
File "/home/user/.local/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/home/user/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/user/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError
ds_report output
[2024-02-27 08:15:56,143] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
[WARNING] sparse_attn is not compatible with ROCM
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/user/.local/lib/python3.10/site-packages/torch']
torch version .................... 2.1.1+rocm5.6
deepspeed install path ........... ['/home/user/.local/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.13.4, unknown, unknown
torch cuda version ............... None
torch hip version ................ 5.6.31061-8c743ae5d
nvcc version ..................... None
deepspeed wheel compiled w. ...... torch 2.1, hip 5.6
shared memory (/dev/shm) size .... 64.00 GB
Docker context
rocm/dev-ubuntu-22.04:5.6.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the provided minimal.py example with Microsoft Phi-2 and tensor parallelism. Read deepspeed/module_inject/layers.py around line 80 and the inference path in deepspeed/inference/engine.py, then compare the injected layer dimensions with the Phi model trace. Done means generation completes without the reported matrix shape error for the reproduced configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100