How to support TheBloke/Falcon-180B-Chat-GPTQ
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
I've tried [docs/gptq.md](https://github.com/lm-sys/FastChat/blob/main/docs/gptq.md), however it throws RuntimeError Error(s) in loading state_dict for LlamaForCausalLM, maybe the difference between different model architectures. Is there any other methods to load this quantisation model?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/gptq.md and reproduce the reported state_dict loading error for TheBloke/Falcon-180B-Chat-GPTQ. Compare the Falcon model architecture with the LlamaForCausalLM path and determine whether a supported loading method can be defined; done means the quantized model loads successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100