After training vicuna-13b, can not find usual 'pytorch_model-00001-of-00003.bin' file in the output dir, only contains the pytorch_model.bin with small size. Looks like the model need post proceess...

Nobody has claimed this yet.

Hi, I try to use deepspeed to train vicuna-13b model with train_mem.py.
Here is my code.

`source activate vicuna
export PYTHONPATH=/data/zhangruixi/FastChat/
deepspeed /data/zhangruixi/FastChat/fastchat/train/train_mem.py \
--model_name_or_path /data/model/vicuna-13b-v1.3/ \
--data_path /data/zhangruixi/FastChat/playground/data/Otto\ Hegde.json \
--bf16 True \
--output_dir /data-xgb1/output/vicuna-13b-otto-ruixi-stage3 \
--num_train_epochs 1 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--gradient_accumulation_steps 2 \
--evaluation_strategy "steps" \
--save_strategy "steps" \
--eval_steps 300 \
--save_steps 300 \
--save_total_limit 1 \
--weight_decay 0. \
--logging_steps 1 \
--tf32 True \
--model_max_length 4096 \
--lazy_preprocess True \
--deepspeed /data/zhangruixi/FastChat/playground/deepspeed_config_s3.json `

And this is my output dir content:

170G vicuna-13b-otto-ruixi-stage3/checkpoint-1600
4.0K vicuna-13b-otto-ruixi-stage3/config.json
4.0K vicuna-13b-otto-ruixi-stage3/generation_config.json
49G vicuna-13b-otto-ruixi-stage3/pytorch_model.bin
272K vicuna-13b-otto-ruixi-stage3/runs
4.0K vicuna-13b-otto-ruixi-stage3/special_tokens_map.json
4.0K vicuna-13b-otto-ruixi-stage3/tokenizer_config.json
492K vicuna-13b-otto-ruixi-stage3/tokenizer.model
168K vicuna-13b-otto-ruixi-stage3/trainer_state.json
8.0K vicuna-13b-otto-ruixi-stage3/training_args.bin

I have tried with
https://github.com/lm-sys/FastChat/blob/55051ad0f23fef5eeecbda14a2e3e128ffcb2a98/fastchat/utils.py#L166-L185
and dose not seem help.

While the `zero_to_fp32.py` inside of folder`checkpoint-1600` seems help, but as I understand, it only convert the model from checkpoint instead of the final trained model.

Start with fastchat/train/train_mem.py and the referenced fastchat/utils.py lines, then inspect the supplied DeepSpeed configuration and checkpoint-1600/zero_to_fp32.py. Reproduce the save behavior and compare the root output with the converted checkpoint; done means the expected final trained weights are identified and the output or conversion path is clarified.

Written by the indexing model from the issue text.