[BUG] AttributeError: module 'deepspeed' has no attribute 'init_inference'

Loading checkpoint shards: 100%|██████████████████| 3/3 [00:03<00:00, 1.09s/it] Traceback (most recent call last): File "/media/mldadmin/home/s123mdg31_11/HeteroDefense/deepspeed_demo.py", line 34, in <module> llm_pipe = init_llm() File "/media/mldadmin/home/s123mdg31_11/HeteroDefense/deepspeed_demo.py", line 16, in init_llm model = deepspeed.init_inference( AttributeError: module 'deepspeed' has no attribute 'init_inference'
Hi guys, when I want to use deepspeed to speed up my code, I met the the above error. I don't get the reason. My deepspeed version is 0.16.4. I run the code in linux. Below is my code:

`import torch
import deepspeed
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

def init_llm():
model = AutoModelForCausalLM.from_pretrained(
"facebook/opt-13b",
torch_dtype=torch.float16,
device_map="auto"
)

model = deepspeed.init_inference(
    model,
    mp_size=1,  
    dtype=torch.float16,
    replace_method="auto",  
    replace_with_kernel_inject=True  
)
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-13b")
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
return pipe

if name == "main":
llm_pipe = init_llm()
output = llm_pipe("Who are you?", max_new_tokens=50)
print(output)
`

May I know how to address that?

Start by reproducing the import and AttributeError in deepspeed_demo.py, focusing on init_llm and the reported DeepSpeed 0.16.4 environment. Check how the installed package exposes inference initialization and verify the result with the facebook/opt-13b loading path; done means the reported setup has a documented, reproducible resolution.