Lightning-AI / Lightning-AI/litgpt
Can we integrating LongRoPE.
Open
enhancement
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
They have implemented LongRope patch for llama and mistral. Is it possible to port same into ligGPT ?
https://github.com/datamllab/LongLM/tree/master
Contributor guide
Research direction
Start by reading the linked LongLM LongRoPE implementation and comparing its llama and mistral patches with litGPT’s model integration points. Done means defining the supported scope and validating LongRoPE for the selected litGPT models; the issue names no files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100