How to keep the model loaded in memory?

Hi guys,

I'm trying to setup a local instance of Phi-2 to use it as an autocomplete provider for my text editor.

The problem that I have is that each time I call the command to complete a text, the files have to be retrieved and the model loaded - which is a lot of time wasted for real time autocompletion.

`/.../candle/target/release/examples$ ./phi --model 2 --quantized --sample-len 12 --prompt "$(cat text-to-complete.md)"`

avx: false, neon: true, simd128: false, f16c: false
temp: 0.00 repeat-penalty: 1.10 repeat-last-n: 64
retrieved the files in 455.042µs
loaded the model in 2.127639167s
starting the inference loop
# The World History

Have you ever wondered how people lived in the past? ...

Do you know how to keep the model loaded in memory?
Like... Is there a possibility to start a server accepting post requests with prompts to complete - or something like this?

Thanks

Start with the `target/release/examples/phi` entry point shown in the report and trace how it retrieves files, loads the model, and enters inference. Clarify whether the desired result is a persistent process or a POST server, then define done as repeated prompts reusing the loaded model without repeating startup work.

Written by the indexing model from the issue text.