Script utilizing LLM

Can you provide a script similar to ``inference-example.py``, that utilises ``run_generation.py`` file? i.e instead of command like execution
``python src/run_generation.py --model_type llama --model_name_or_path meta-llama/Llama-2-13b-chat-hf \
--prefix "[INST] <>\n You are a helpful assistant. Answer with detailed responses according to the entire instruction or question. \n<>\n\n Summarize the following book: " \
--prompt example_inputs/harry_potter_full.txt \
--suffix " [/INST]" --test_unlimiformer --fp16 --length 200 --layer_begin 16 \
--index_devices 1 --datastore_device 1 ``
instead load the model and run inference from python script.
Thanks in advance!

Look at src/run_generation.py to understand how the model is loaded and inference is run. The script should import the necessary modules, load the model similarly to the command line, and run generation programmatically. Check example_inputs/harry_potter_full.txt for the input format. The goal is to produce a script that replicates the command line functionality but is callable from another Python script.

由索引模型根據 Issue 內容生成。