deepspeedai / deepspeedai/DeepSpeed

[BUG] how to parse the parameters of the respective section using hydra and argparse?

Open
#5,357 3 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug training
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

I'm training unet with deepspeed.

The basic trainging code parses parameters through hydra. The simplified code is

@hydra.main(version_base="1.2", config_path="config", config_name="config_rdb")
def main(cfg:DictConfig):
    print(OmegaConf.to_yaml(cfg))
    run_training_Unet(
         if_training=cfg.args.if_training,
         epochs=cfg.args.epochs,
        )

if __name__=="__main__":
    main()
python train.py +args=config.yaml ++args.modename='unet'

I change the basica code to deepseep version.

@hydra.main(version_base="1.2", config_path="config", config_name="config_rdb")
def main(cfg:DictConfig, args,):
    print(OmegaConf.to_yaml(cfg))
    run_training_Unet(
         if_training=cfg.args.if_training,
         epochs=cfg.args.epochs,
         args=args
     )

if __name__=="__main__":
    parser=argparse.ArgumentParser(description='deepspeed')
    parser.add_argument('--deepspeed_config',type=str,default=None,
                           help='Path')
    parser.add_argument('--local_rank',type=int,default=None,
                            help='local rank for deepspeed')
    parser.add_argument('--global_rank', type=int, default=None, 
					help='global rank')
    args=parser.parse_args()
    main(args)
deepspeed train.py +args=config.yaml ++args.modename='unet' --deepspeed --deepspeed_config ds_config.json

But i got a error.

[2024-03-30 18:41:09,000] [ERROR] [launch.py:184:sigkill_handler] ['/public/home/y/bin/python', '-u', 'train.py', '--local_rank=0', '+args=config.yaml', '++args.model_name=Unet', '--deepspeed', '--deepspeed_config', 'ds_config.json'] exits with return code = 2
usage: train.py [--help] [--hydra-help] [--version]
                                         [--cfg {job,hydra,all}] [--resolve]
                                         [--package PACKAGE] [--run]
                                         [--multirun] [--shell-completion]
                                         [--config-path CONFIG_PATH]
                                         [--config-name CONFIG_NAME]
                                         [--config-dir CONFIG_DIR]
                                         [--experimental-rerun EXPERIMENTAL_RERUN]
                                         [--info [{all,config,defaults,defaults-tree,plugins,searchpath}]]
                                         [overrides ...]
train_models_forward_deepspeed.py: error: unrecognized arguments: --local_rank=0

I think it's the conflict of hydra and argparse parsing parameters. Is there a tutorial code?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the train.py entry point, the @hydra.main function, and the argparse setup shown in the report. Reproduce the DeepSpeed command and its --local_rank=0 error, then compare how Hydra and argparse receive arguments. Done should be a documented, working parameter-parsing pattern or a confirmed correction to the invocation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.