Significantly different results with inference using a saved checkpoint v/s inferencing during fine-tuning

Nobody has claimed this yet.

I am using lora for fine-tuning Llama-2-7b-chat-hf with my custom dataset. This is the command I am running:
```!CUDA_VISIBLE_DEVICES=2,3 python finetune/lora.py --data_dir '../new_test_data_llm' --checkpoint_dir '../checkpoint/Llama-2-7b-chat-hf/' --out_dir '../output_chat_model_temp'```

Loss is decreasing steadily:
```iter 90 step 91: loss 0.4730, iter time: 2164.59ms (optimizer.step)
iter 91 step 92: loss 0.3682, iter time: 2216.19ms (optimizer.step)
iter 92 step 93: loss 0.5185, iter time: 2389.64ms (optimizer.step)
iter 93 step 94: loss 0.3875, iter time: 2417.41ms (optimizer.step)
iter 94 step 95: loss 0.1968, iter time: 2399.70ms (optimizer.step)
iter 95 step 96: loss 0.2645, iter time: 2209.37ms (optimizer.step)
iter 96 step 97: loss 0.3107, iter time: 2206.34ms (optimizer.step)
iter 97 step 98: loss 0.3512, iter time: 2174.37ms (optimizer.step)
iter 98 step 99: loss 0.3052, iter time: 2338.01ms (optimizer.step)
iter 99 step 100: loss 0.3454, iter time: 2274.14ms (optimizer.step)
```
and validation step that runs after 100 iters also shows that the response is shaping up correctly (at least generating the right response format in terms of number of words etc). This is the output from inferencing after 100 iterations when getting triggered from fine-tuning script (finetune/lora.py).
```
iter 98 step 99: loss 0.3051, iter time: 2506.50ms (optimizer.step)
iter 99 step 100: loss 0.3453, iter time: 2524.91ms (optimizer.step)
Validating ...
Rephrase the ... ... request.

### Instruction:
Rephrase ... ... crisp response.

### Input:
0 year girls frock

### Response:8 year girl clothess
```
But later when I run seperate inference call with explicit loading of the checkpoint (trained from above) with this command:
```!python generate/lora.py --checkpoint_dir '../LLM/checkpoint/Llama-2-7b-chat-hf/' --lora_path '../output_chat_model_temp/lit_model_lora_finetuned.pth' --prompt 'Rephrase ... ... crisp response.' --input '0 year girls frock'```

I get this weird output:
```Loading model '/var/lib/search-data/abhinav.a/LLM/checkpoint/Llama-2-7b-chat-hf/lit_model.pth' with {'org': 'meta-llama', 'name': 'Llama-2-7b-chat-hf', 'block_size': 4096, 'vocab_size': 32000, 'padding_multiple': 64, 'padded_vocab_size': 32000, 'n_layer': 32, 'n_head': 32, 'n_embd': 4096, 'rotary_percentage': 1.0, 'parallel_residual': False, 'bias': False, 'lm_head_bias': False, 'n_query_groups': 32, 'shared_attention_norm': False, '_norm_class': 'RMSNorm', 'norm_eps': 1e-05, '_mlp_class': 'LLaMAMLP', 'gelu_approximate': 'none', 'intermediate_size': 11008, 'rope_condense_ratio': 1, 'rope_base': 10000, 'r': 8, 'alpha': 16, 'dropout': 0.05, 'to_query': True, 'to_key': False, 'to_value': True, 'to_projection': False, 'to_mlp': False, 'to_head': False, 'head_size': 128, 'rope_n_elem': 128}
Time to instantiate model: 3.37 seconds.
Time to load the model weights: 8.97 seconds.
Number of total parameters: 6,742,609,920
0 years girls females andmature 3 years girls baby carriage andmature 5 years girls females andmature 7 years girls females and 9 years girls females andmature 11 years girls females, etc.
0 years girls females and 0 years girls females 0 years girls females, etc.
0 years girls females, etc.

### Instruction:
Repeat the story2018 girls females, etc.
0 years girls

Time for inference: 4.89 sec total, 20.46 tokens/sec
Memory used: 13.82 GB
```

Why is this happening? Can someone please point it if I am doing something wrong here. I think the training is going on fine (evident with decreasing loss and intermediate correct responses in validation steps), but final inferencing output is so off, not able to understand why.

Compare the inference path in finetune/lora.py with generate/lora.py, including how the checkpoint_dir and lora_path are loaded and how the prompt and input are constructed. Reproduce the validation output after 100 iterations, then run the separate command and identify the first loading or generation difference; done means both paths produce consistent results from the same saved checkpoint.

Written by the indexing model from the issue text.