lm-sys / lm-sys/FastChat

Make chatglm2-6b load8bit work on Mac m2 with mps(fix bfloatxx error)

Open
#2,295 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

First,I use ` python -m fastchat.serve.cli --model-path /my/mac/path/llm_models/chatglm2-6b --load-8bit --device mps`
to run chatglm2-6b.But I got this error: `Trying to convert BFloat16 to the MPS backend but it does not have support for that dtype`.

I guess I should make it to float from bfloat.I try to add .half() to the source code of Fastchat. And It works.

fastchat/model/compress.py
```
for filename in tqdm(files):
tmp_state_dict = torch.load(filename, map_location=lambda storage, loc: storage)
for name in tmp_state_dict:
if name in linear_weights:
tensor = tmp_state_dict[name].half().to(device).data.to(torch_dtype)
compressed_state_dict[name] = compress(
tensor, default_compression_config
)
else:
compressed_state_dict[name] = tmp_state_dict[name].half().to(device)
tmp_state_dict[name] = None
tensor = None
gc.collect()
torch.cuda.empty_cache()
if device == "xpu":
torch.xpu.empty_cache()
````
image

Maybe we should add an arg for doing this `half()` operator?

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with the FastChat CLI command using chatglm2-6b, --load-8bit, and --device mps. Read fastchat/model/compress.py around the state-dict loading and conversion shown in the report, then determine how the bfloat16 conversion should be handled for MPS without changing other devices. Done means the model loads on a Mac M2 without the reported BFloat16 MPS error.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.