Provide HF AutoModel interface
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 536
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 4
Description
I am interested in adding CTranslate2 support to LMQL similar to this PR which added autogptq support. In reality, it seems like the bulk of the work would have to be done in CTranslate2 to provide a high level HF Transformers "CTranslate2ForCausalLM" model interface.
I understand that the ctranslate2.Generator already provides very high level interfaces for language model text generation, but by providing the HF Transformers interface, CTranslate2 could be easily added to many more frameworks like LMQL and Microsoft's Guidance.
What do you think? Thank 😁
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing ctranslate2.Generator interface and the autogptq integration PR linked in the issue. The payload names no implementation files or tests; done would require a defined CTranslate2ForCausalLM interface that can be consumed by Hugging Face Transformers and downstream frameworks such as LMQL or Guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, huggingface
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100