deepspeedai / deepspeedai/DeepSpeedExamples

AttributeError: 'Namespace' object has no attribute 'tensor_model_parallel_size'

Open
#94 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
6.8k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
1

Description

I am trying to run the GPT2 model on Google Colab. I have placed my training data in a loose JSON format, with one JSON containing a text sample per line. On running the preprocessing script, I am getting the above error. Kindly help me with this.

complete error below:

Opening my-corpus.json

building GPT2BPETokenizer tokenizer ...
Traceback (most recent call last):
File "/content/DeepSpeedExamples/Megatron-LM-v1.1.5-ZeRO3/tools/preprocess_data.py", line 200, in
main()
File "/content/DeepSpeedExamples/Megatron-LM-v1.1.5-ZeRO3/tools/preprocess_data.py", line 154, in main
tokenizer = build_tokenizer(args)
File "/usr/local/lib/python3.7/dist-packages/megatron/tokenizer/tokenizer.py", line 48, in build_tokenizer
args)
File "/usr/local/lib/python3.7/dist-packages/megatron/tokenizer/tokenizer.py", line 59, in _vocab_size_with_padding
args.tensor_model_parallel_size
AttributeError: 'Namespace' object has no attribute 'tensor_model_parallel_size'

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tools/preprocess_data.py around main() and build_tokenizer, then inspect megatron/tokenizer/tokenizer.py at _vocab_size_with_padding where the traceback occurs. Reproduce the reported GPT-2 preprocessing run in Google Colab and verify that it completes without the tensor_model_parallel_size AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.