microsoft / microsoft/onnxruntime

Support for GPT-Neo and GPT-J models

Open
#10,196 4 comments 4 reactions 1 assignee View on GitHub

@faxu is already working on this.

Since Jan 5, 2022.

feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 8h
Merged PRs (30d)
179

Description

Is your feature request related to a problem? Please describe.
I see quite some GPT-2 implementations, but I am missing a GPT-Neo/GPT-J implementation. This is a variant of GPT-2 which is quite similar to GPT-2 in concept, but with much better results compared to GPT-2. Documentation about GPT-Neo can be found here: https://huggingface.co/docs/transformers/model_doc/gpt_neo

System information

  • ONNX Runtime version (you are using): 1.11.0

Describe the solution you'd like
A demo that shows that GPT-Neo and GPT-J models can be used with ONNX to speed up huggingface models.

Describe alternatives you've considered
Since GPT-2 with LMhead outputs a similar output as GPTNeoForCausalLM, it should be possible to clone the demo. However, optimization will need to be done differently.

Additional context
I am one of the contributors behind KoboldAI, an OSS application that uses GPT-Neo and GPT-J models for creating short novels and text. Am looking for ways to optimize and speed up the models we distribute.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.