[Feature Request] Add a ctranslate2 model worker
@fozziethebeat is already working on this.
Since Aug 2, 2023.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
According to some [recent analysis](https://twitter.com/HamelHusain/status/1685074309549858816) on twitter, [CTranslate2](https://github.com/OpenNMT/CTranslate2) can serve LLMs a little faster than vLLM and (maybe?) with a small quality increase. At least for Llama 2.
This could either be a a model worker that's added directly to fastchat OR a doc with extensive documentation on how to write a custom model worker (with mostly working \ implementation code) that anyone can use in their own project. The second option might be best so that people can expand FastChat with custom model workers without having to change the base project too much.
If this is appealing I can get to it at some point.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.