RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'

Nobody has claimed this yet.

When using cpu-offloading on my Tesla P4 8GB GPU i error out.

gnome@ailab:~$ **python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.3 --load-8bit --cpu-offloading**
```
You are using the default legacy behaviour of the . If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:05<00:00, 2.82s/it]
USER: hi
ASSISTANT: Traceback (most recent call last):
File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.9/dist-packages/fastchat/serve/cli.py", line 283, in
main(args)
File "/usr/local/lib/python3.9/dist-packages/fastchat/serve/cli.py", line 208, in main
chat_loop(
File "/usr/local/lib/python3.9/dist-packages/fastchat/serve/inference.py", line 479, in chat_loop
outputs = chatio.stream_output(output_stream)
File "/usr/local/lib/python3.9/dist-packages/fastchat/serve/cli.py", line 61, in stream_output
for outputs in output_stream:
File "/usr/local/lib/python3.9/dist-packages/torch/utils/_contextlib.py", line 35, in generator_context
response = gen.send(None)
File "/usr/local/lib/python3.9/dist-packages/fastchat/serve/inference.py", line 123, in generate_stream
out = model(torch.as_tensor([input_ids], device=device), use_cache=True)
File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/transformers/models/llama/modeling_llama.py", line 820, in forward
outputs = self.model(
File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/transformers/models/llama/modeling_llama.py", line 708, in forward
layer_outputs = decoder_layer(
File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/transformers/models/llama/modeling_llama.py", line 424, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/transformers/models/llama/modeling_llama.py", line 321, in forward
query_states = self.q_proj(hidden_states)
File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'
```

Start with fastchat/serve/cli.py and fastchat/serve/inference.py, especially the generate_stream path shown in the traceback. Reproduce the documented command with --load-8bit and --cpu-offloading, then trace where Half tensors reach the CPU linear operation. Done means the command can generate a response without the addmm_impl_cpu_ error.

Written by the indexing model from the issue text.