deepspeedai / deepspeedai/DeepSpeed

[BUG]binascii.Error: Incorrect padding

Open
#5,378 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug training
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Describe the bug
A clear and concise description of what the bug is.
When I running the code deepspeed --num_nodes 1 --num_gpus 4 run_glue_classifier_bert_large.py --task_name RTE --do_train --do_eval --deepspeed --preln --deepspeed_config pwd/glue_bert_large.json --do_lower_case --data_dir /workspace/glue_data/RTE --bert_model bert-large-cased --max_seq_length 128 --train_batch_size 4 --gradient_accumulation_steps 1 --learning_rate 2e-5 --num_train_epochs 3 --output_dir ./output/bert_RTE --model_file 0
To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error
    Traceback (most recent call last):
    File "/workspace/DeepSpeedExamples/training/BingBertGlue/run_glue_classifier_bert_large.py", line 1260, in
    main()
    File "/workspace/DeepSpeedExamples/training/BingBertGlue/run_glue_classifier_bert_large.py", line 961, in main
    model, optimizer, _, _ = deepspeed.initialize(
    File "/opt/conda/lib/python3.10/site-packages/deepspeed/init.py", line 157, in initialize
    config_class = DeepSpeedConfig(config, mpu)
    File "/opt/conda/lib/python3.10/site-packages/deepspeed/runtime/config.py", line 706, in init
    config_decoded = base64.urlsafe_b64decode(config).decode('utf-8')
    File "/opt/conda/lib/python3.10/base64.py", line 133, in urlsafe_b64decode
    return b64decode(s)
    File "/opt/conda/lib/python3.10/base64.py", line 87, in b64decode
    return binascii.a2b_base64(s)
    Expected behavior
    A clear and concise description of what you expected to happen.

ds_report output
Please run ds_report to give us details about your setup.
[2024-04-08 16:59:46,458] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 7077
[2024-04-08 16:59:46,460] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 7078
[2024-04-08 16:59:46,461] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 7079
[2024-04-08 16:59:46,461] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 7080
[2024-04-08 16:59:46,463] [ERROR] [launch.py:321:sigkill_handler] ['/opt/conda/bin/python', '
Screenshots
If applicable, add screenshots to help explain your problem.

System info (please complete the following information):

  • OS: [Ubuntu 20.04]
  • GPU count and types [e.g. one machines with 4*NVIDIA RTX A5000 ]
  • CUDA Version: 11.7
  • Pytorch version 2.0.1
  • Any other relevant info about your setup
    Alex

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?

Docker context
Are you using a specific docker image that you can share?
2.0.1-cuda11.7-cudnn8
Additional context
Add any other context about the problem here.

### Tasks

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with deepspeed/runtime/config.py at DeepSpeedConfig and the invocation in run_glue_classifier_bert_large.py, especially the --deepspeed_config argument. Reproduce the command with the supplied environment details and collect ds_report; done means the RTE training command loads its configuration without the binascii padding error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.