linkedin / linkedin/Liger-Kernel
Support the new Solar architecture
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.6k
- Forks
- 603
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 47
Description
### 🚀 The feature, motivation and pitch
This model from Upstage is extremely strong for models that fit on a single GPU for training and inference! https://huggingface.co/upstage/solar-pro-preview-instruct. However, it does use a custom architecture `solar` which is based on Llama/Mistral but modifies the forward pass to add long range residual connections. It would be awesome to support this architecture natively out of the box!
### Alternatives
_No response_
### Additional context
Thank you so much for this awesome project :)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing Llama/Mistral architecture support and the linked Solar model description. Identify where Solar's custom long-range residual forward pass must be integrated; done means the Solar architecture works natively for training and inference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100