Support peft's LoRa for HF transformer models.
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 536
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 4
Description
Context: With HF models, one can use peft to do parameter efficient tuning, the most popular (and afaik most performant) method being LoRa.
Idea: It would be great to be able to have an instance (in GPU memory) of a base HF transformer model (running with CT2) that you run with multiple instances of of LoRa weights.
Would be curious to hear if you think this could be done in CT2 in a generic way that's applicable to all HF transformer models (just like HF's peft).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the linked peft project and CTranslate2's existing support for Hugging Face transformer models. Clarify whether one base model in GPU memory can serve multiple LoRa weight instances generically across HF models; done should include an agreed design and demonstrated support for that workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100