deepspeedai / deepspeedai/DeepSpeedExamples
BERT in Megatron-LM-v1.1.5-3D_parallelism does not support pipeline parallelism
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
I try to run the BERT with pipeline parallelism, but I get an error:
File "DeepSpeedExamples/Megatron-LM-v1.1.5-3D_parallelism/pretrain_bert.py", line 146, in
args_defaults={'tokenizer_type': 'BertWordPieceLowerCase'})
File "/DeepSpeedExamples/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 81, in pretrain
model, optimizer, lr_scheduler = setup_model_and_optimizer(model_provider)
File "/DeepSpeedExamples/Megatron-LM-v1.1.5-3D_parallelism/megatron/training.py", line 252, in setup_model_and_optimizer
model.set_batch_fn(model.module._megatron_batch_fn)
File "/home/wwu/anaconda3/envs/sospx86/lib/python3.6/site-packages/torch/nn/modules/module.py", line 948, in getattr
type(self).name, name))
AttributeError: 'DeepSpeedEngine' object has no attribute 'set_batch_fn'
I dig into the code a little bit, it seems like the pipeline parallelism is not implemented for BERT.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing pretrain_bert.py into megatron/training.py, especially setup_model_and_optimizer and the set_batch_fn call shown in the traceback. Compare the BERT path with the pipeline-parallel path to identify the missing support; done means BERT runs with pipeline parallelism without the DeepSpeedEngine AttributeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100