RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'

Nobody has claimed this yet.

Running my code on collab GPU but getting this error, can someone please help:

Traceback (most recent call last):
File "/content/drive/MyDrive/Self_Reflection_Medical-main/FastChat/loop.py", line 295, in
final_knowledge, final_response, all_history_knowledge, all_history_response = main_loop(args, line, model, tokenizer, knowledge_loop, response_loop)
File "/content/drive/MyDrive/Self_Reflection_Medical-main/FastChat/loop_utils.py", line 15, in main_loop
final_knowledge, history_knowledge = knowledge_loop(args, model, tokenizer, question)
File "/content/drive/MyDrive/Self_Reflection_Medical-main/FastChat/loop.py", line 94, in knowledge_loop
knowledge = generate_step(args, model, tokenizer, prompt, conv)
File "/content/drive/MyDrive/Self_Reflection_Medical-main/FastChat/loop.py", line 45, in generate_step
for outputs in generate_stream( model, tokenizer, params, args.device,context_len=2048):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 35, in generator_context
response = gen.send(None)
File "/content/drive/MyDrive/Self_Reflection_Medical-main/FastChat-main/fastchat/serve/inference.py", line 132, in generate_stream
out = model(input_ids=start_ids, use_cache=True)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py", line 1176, in forward
outputs = self.model(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py", line 1019, in forward
layer_outputs = decoder_layer(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py", line 740, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py", line 352, in forward
query_states = self.q_proj(hidden_states)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'

Start with loop.py at generate_step and knowledge_loop, then follow loop_utils.py into fastchat/serve/inference.py at generate_stream. Reproduce the reported Colab GPU path and inspect the model, input IDs, args.device, and tensor dtype when the Llama forward call begins. Done means the device and dtype mismatch is identified and generation completes without this runtime error.

Written by the indexing model from the issue text.