RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:

Nobody has claimed this yet.

Traceback (most recent call last):
File "/data/app/FastChat/fastChat/train/train.py", line 335, in
train()
File "/data/app/FastChat/fastChat/train/train.py", line 328, in train
trainer.train(resume_from_checkpoint=True)
File "/data/app/install/transformers/src/transformers/trainer.py", line 1651, in train
self._load_from_checkpoint(resume_from_checkpoint)
File "/data/app/install/transformers/src/transformers/trainer.py", line 2159, in _load_from_checkpoint
load_result = model.load_state_dict(state_dict, False)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for LlamaForCausalLM:
size mismatch for model.embed_tokens.weight: copying a param with shape torch.Size([32001, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096]).
size mismatch for lm_head.weight: copying a param with shape torch.Size([32001, 4096]) from checkpoint, the shape in current model is torch.Size([32000, 4096]).
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 6660) of binary: /usr/bin/python3

CUDA:12.1
torch:11.8
transformers 4.28-dev

Start with fastChat/train/train.py at train() and the resume_from_checkpoint=True call, then inspect transformers Trainer._load_from_checkpoint and the reported LlamaForCausalLM state-dict shapes. Reproduce the checkpoint-loading failure with the listed Python, torch, and Transformers versions. Done means resuming training no longer raises the embedding and lm_head size mismatch.

Written by the indexing model from the issue text.