Load finetune model fails

I've fine-turned llama-7b model with command:

#!/bin/bash

deepspeed_args="--num_gpus=8 --master_port=11000"

exp_id=llama-7b-v2
project_dir=XXXX
base_model_path=${project_dir}/models/pinkmanlove/llama-7b-hf
lora_model_path=${project_dir}/models/llama7b-lora-380k

output_dir=${project_dir}/output_models/${exp_id}
log_dir=${project_dir}/log/${exp_id}

dataset_path=${project_dir}/dataset/train_2M_CN/lmflow/

mkdir -p ${output_dir} ${log_dir}

deepspeed ${deepspeed_args} \
  finetune.py \
    --model_name_or_path ${base_model_path} \
    --lora_model_path ${lora_model_path} \
    --dataset_path ${dataset_path} \
    --output_dir ${output_dir} --overwrite_output_dir \
    --num_train_epochs 2 \
    --learning_rate 1e-4 \
    --block_size 512 \
    --per_device_train_batch_size 1 \
    --use_lora 1 \
    --lora_r 10 \
    --deepspeed configs/ds_config_zero3.json \
    --run_name finetune_with_lora \
    --validation_split_percentage 0 \
    --logging_steps 20 \
    --do_train \
    --ddp_timeout 72000 \
    --save_steps 5000 \
    --dataloader_num_workers 8 \
    | tee ${log_dir}/train.log \
    2> ${log_dir}/train.err

The output of the fine-turned model is: ${project_dir}/output_models/llama-7b-v2, it generates checkpoints as:

.
├── adapter_config.json
├── adapter_model.bin
├── all_results.json
├── checkpoint-25000
│   ├── global_step25000
│   │   ├── zero_pp_rank_0_mp_rank_00_model_states.pt
│   │   ├── zero_pp_rank_0_mp_rank_00_optim_states.pt
│   │   ├── zero_pp_rank_1_mp_rank_00_model_states.pt
│   │   ├── zero_pp_rank_1_mp_rank_00_optim_states.pt
│   │   ├── zero_pp_rank_2_mp_rank_00_model_states.pt
│   │   ├── zero_pp_rank_2_mp_rank_00_optim_states.pt
│   │   ├── zero_pp_rank_3_mp_rank_00_model_states.pt
│   │   ├── zero_pp_rank_3_mp_rank_00_optim_states.pt
│   │   ├── zero_pp_rank_4_mp_rank_00_model_states.pt
│   │   ├── zero_pp_rank_4_mp_rank_00_optim_states.pt
│   │   ├── zero_pp_rank_5_mp_rank_00_model_states.pt
│   │   ├── zero_pp_rank_5_mp_rank_00_optim_states.pt
│   │   ├── zero_pp_rank_6_mp_rank_00_model_states.pt
│   │   ├── zero_pp_rank_6_mp_rank_00_optim_states.pt
│   │   ├── zero_pp_rank_7_mp_rank_00_model_states.pt
│   │   └── zero_pp_rank_7_mp_rank_00_optim_states.pt
│   ├── latest
│   ├── pytorch_model.bin
│   ├── rng_state_0.pth
│   ├── rng_state_1.pth
│   ├── rng_state_2.pth
│   ├── rng_state_3.pth
│   ├── rng_state_4.pth
│   ├── rng_state_5.pth
│   ├── rng_state_6.pth
│   ├── rng_state_7.pth
│   ├── special_tokens_map.json
│   ├── tokenizer_config.json
│   ├── tokenizer.model
│   ├── trainer_state.json
│   ├── training_args.bin
│   └── zero_to_fp32.py
├── README.md
├── trainer_state.json
└── train_results.json

When I tried to load model with command

./scripts/run_chatbot.sh \
     ${project_dir}/output_models/llama-7b-v2 \
    ${project_dir}/models/llama7b-lora-380k

I complains OSError: ${project_dir}/output_models/llama-7b-v2 does not appear to have a file named config.json. Checkout 'https://huggingface.co/${project_dir}//output_models/llama-7b-v2/None' for available files..
However followed the suggestion 290 to load model with command

./scripts/run_chatbot.sh \
     ${project_dir}/models/pinkmanlove/llama-7b-hf/ \
     ${project_dir}/output_models/llama-7b-v2/

I got the error log:

Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:10<00:00,  5.18s/it]
Traceback (most recent call last):
  File "/apdcephfs/share_698083/xishengzhao/LMFlow/examples/chatbot.py", line 159, in <module>
    main()
  File "/apdcephfs/share_698083/xishengzhao/LMFlow/examples/chatbot.py", line 73, in main
    model = AutoModel.get_model(
  File "/LMFlow/src/lmflow/models/auto_model.py", line 14, in get_model
    return HFDecoderModel(model_args, *args, **kwargs)
  File "/LMFlow/src/lmflow/models/hf_decoder_model.py", line 219, in __init__
    self.backend_model = PeftModel.from_pretrained(
  File "/venv/lib/python3.9/site-packages/peft/peft_model.py", line 161, in from_pretrained
    model = set_peft_model_state_dict(model, adapters_weights)
  File "/venv/lib/python3.9/site-packages/peft/utils/save_and_load.py", line 74, in set_peft_model_state_dict
    model.load_state_dict(peft_model_state_dict, strict=False)
  File "/venv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for PeftModelForCausalLM:
        size mismatch for base_model.model.model.layers.0.self_attn.q_proj.lora_A.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([10, 4096]).
        size mismatch for base_model.model.model.layers.0.self_attn.q_proj.lora_B.weight: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([4096, 10]).

Any suggestion for correctly loading fine-turned models. Much appreciate!

Reproduce the loading commands from scripts/run_chatbot.sh, then inspect examples/chatbot.py and src/lmflow/models/hf_decoder_model.py around PeftModel.from_pretrained. Compare the saved adapter files and configuration with the base model and verify that a corrected finetuned checkpoint loads successfully without the reported state-dict mismatch.