microsoft / microsoft/onnxruntime
Support for GPT-Neo and GPT-J models
@faxu is already working on this.
Since Jan 5, 2022.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 8h
- Merged PRs (30d)
- 179
Description
Is your feature request related to a problem? Please describe.
I see quite some GPT-2 implementations, but I am missing a GPT-Neo/GPT-J implementation. This is a variant of GPT-2 which is quite similar to GPT-2 in concept, but with much better results compared to GPT-2. Documentation about GPT-Neo can be found here: https://huggingface.co/docs/transformers/model_doc/gpt_neo
System information
- ONNX Runtime version (you are using): 1.11.0
Describe the solution you'd like
A demo that shows that GPT-Neo and GPT-J models can be used with ONNX to speed up huggingface models.
Describe alternatives you've considered
Since GPT-2 with LMhead outputs a similar output as GPTNeoForCausalLM, it should be possible to clone the demo. However, optimization will need to be done differently.
Additional context
I am one of the contributors behind KoboldAI, an OSS application that uses GPT-Neo and GPT-J models for creating short novels and text. Am looking for ways to optimize and speed up the models we distribute.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.