Make chatglm2-6b load8bit work on Mac m2 with mps(fix bfloatxx error)
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
First,I use ` python -m fastchat.serve.cli --model-path /my/mac/path/llm_models/chatglm2-6b --load-8bit --device mps`
to run chatglm2-6b.But I got this error: `Trying to convert BFloat16 to the MPS backend but it does not have support for that dtype`.
I guess I should make it to float from bfloat.I try to add .half() to the source code of Fastchat. And It works.
fastchat/model/compress.py
```
for filename in tqdm(files):
tmp_state_dict = torch.load(filename, map_location=lambda storage, loc: storage)
for name in tmp_state_dict:
if name in linear_weights:
tensor = tmp_state_dict[name].half().to(device).data.to(torch_dtype)
compressed_state_dict[name] = compress(
tensor, default_compression_config
)
else:
compressed_state_dict[name] = tmp_state_dict[name].half().to(device)
tmp_state_dict[name] = None
tensor = None
gc.collect()
torch.cuda.empty_cache()
if device == "xpu":
torch.xpu.empty_cache()
````
Maybe we should add an arg for doing this `half()` operator?
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with the FastChat CLI command using chatglm2-6b, --load-8bit, and --device mps. Read fastchat/model/compress.py around the state-dict loading and conversion shown in the report, then determine how the bfloat16 conversion should be handled for MPS without changing other devices. Done means the model loads on a Mac M2 without the reported BFloat16 MPS error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, python, pytorch
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100