deepspeedai / deepspeedai/DeepSpeed
[BUG]binascii.Error: Incorrect padding
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
A clear and concise description of what the bug is.
When I running the code deepspeed --num_nodes 1 --num_gpus 4 run_glue_classifier_bert_large.py --task_name RTE --do_train --do_eval --deepspeed --preln --deepspeed_config pwd/glue_bert_large.json --do_lower_case --data_dir /workspace/glue_data/RTE --bert_model bert-large-cased --max_seq_length 128 --train_batch_size 4 --gradient_accumulation_steps 1 --learning_rate 2e-5 --num_train_epochs 3 --output_dir ./output/bert_RTE --model_file 0
To Reproduce
Steps to reproduce the behavior:
- Go to '...'
- Click on '....'
- Scroll down to '....'
- See error
Traceback (most recent call last):
File "/workspace/DeepSpeedExamples/training/BingBertGlue/run_glue_classifier_bert_large.py", line 1260, in
main()
File "/workspace/DeepSpeedExamples/training/BingBertGlue/run_glue_classifier_bert_large.py", line 961, in main
model, optimizer, _, _ = deepspeed.initialize(
File "/opt/conda/lib/python3.10/site-packages/deepspeed/init.py", line 157, in initialize
config_class = DeepSpeedConfig(config, mpu)
File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/config.py", line 706, in init
config_decoded = base64.urlsafe_b64decode(config).decode('utf-8')
File "/opt/conda/lib/python3.10/base64.py", line 133, in urlsafe_b64decode
return b64decode(s)
File "/opt/conda/lib/python3.10/base64.py", line 87, in b64decode
return binascii.a2b_base64(s)
Expected behavior
A clear and concise description of what you expected to happen.
ds_report output
Please run ds_report to give us details about your setup.
[2024-04-08 16:59:46,458] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 7077
[2024-04-08 16:59:46,460] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 7078
[2024-04-08 16:59:46,461] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 7079
[2024-04-08 16:59:46,461] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 7080
[2024-04-08 16:59:46,463] [ERROR] [launch.py:321:sigkill_handler] ['/opt/conda/bin/python', '
Screenshots
If applicable, add screenshots to help explain your problem.
System info (please complete the following information):
- OS: [Ubuntu 20.04]
- GPU count and types [e.g. one machines with 4*NVIDIA RTX A5000 ]
- CUDA Version: 11.7
- Pytorch version 2.0.1
- Any other relevant info about your setup
Alex
Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?
Docker context
Are you using a specific docker image that you can share?
2.0.1-cuda11.7-cudnn8
Additional context
Add any other context about the problem here.
### Tasks
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with deepspeed/runtime/config.py at DeepSpeedConfig and the invocation in run_glue_classifier_bert_large.py, especially the --deepspeed_config argument. Reproduce the command with the supplied environment details and collect ds_report; done means the RTE training command loads its configuration without the binascii padding error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100