update request on documentation.

Hi,

I'm reading your documentations. Thanks for amazing library. As I'm a little bit confused, I need your help to clarify some questions. My computer has 16 gb memory. Gpu's memory is also 16 gb. I don't have enough source. But I can load a model (For example LLAMA2-7b) with huggingface methods. (Device_map is selected auto. All model is loaded to GPU.) If I try to load this model to cpu, memory is not sufficient.

• I don't want to load model to CPU before calling init_inference. Can I load model to GPU with Deepspeed init_inference? (it will not be loaded to CPU before calling init_inference.) I looked through the examples. You prefer loading model to cpu and then it is loaded to GPU when init_inference is called.
• If I prefer deepspeed.initialize, does it make difference? Before calling deepspeed.initialize, should I still load model to CPU? (with from_pretrained)
• To load model to GPU directly, metadevice can be used. To my knowledge, you're not supporing some models like LLAMA2. (There is a PR but it is not merged) Is there any other way to load model to GPU directly?
• is it possible to use load_checkpoint parameter to load model directly to GPU with deepspeed?
• is there any way to load quantized model with deepspeed. I loaded llama13b with from_pretrained command. load_8bit was enabled. But deepspeed threw error. ( it is not surprising ) Is there any way to quantize model in deepspeed when loading model?
• If I load all parameters to CPU and increase the batch size, should I expect increase in throughput?
• Is there any way to offload only some weights? Because if I offload all weights to CPU, more than %50 of memory is not utilized.

If it doesn't exits, maybe a section can be added to documentation.

Thanks in advance.

Start by reviewing the DeepSpeed inference and initialize documentation and examples, along with Hugging Face from_pretrained, load_checkpoint, quantization, and offloading behavior. Done means providing clear answers to the listed GPU-loading, CPU-memory, quantization, throughput, and partial-offload questions or adding a focused documentation section.