[New Bitnet Model Support Request] Deepgrove model Bonsai 0.5B - Add Channel Scales
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 40.3k
- Forks
- 3.7k
- PR merge metrics
- No merged PRs in 30d
Description
A new SOTA bitnet model, Bonsai 0.5B, has come out. Seems to outperform larger bitnet models like Falcon 1B, 3B, TriLM 700M. Seems like they are going to release a new line of bitnet models which is really exciting.
Support is needed for these models. They adopt a channel wise scaling factor compared to the tensor level ones. Maybe a separate kennel can be built to apply scales outside of the matmul kernels? Probably would yield similar inference speeds. Note that the hugging face does have a custom Q-linear layer that applies the scales.
HF: https://huggingface.co/deepgrove/Bonsai
Seems super promising.
pinging @Eddie-Wang1120 + other kernels writers
Other posts and information:
https://www.reddit.com/r/LocalLLaMA/comments/1jgkqio/new_bitnet_model_from_deepgrove/
https://x.com/deepgrove_ai/status/1903103798735761518
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file or test is named. Start by inspecting the existing tensor-level scaling and matmul kernels, then compare them with the custom Q-linear layer in the Deepgrove/Bonsai Hugging Face model. Done means Bonsai 0.5B is supported with channel-wise scales and its inference behavior is validated against the model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, huggingface
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100