microsoft / microsoft/onnxruntime
[Feature Request] Implement encoder_hidden_states as input in GPT2_BeamSearch Node
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
I try to use the [convert_generation.py](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/convert_generation.py) script to create a GPT2 code generation model with beam search with encoder_hidden_states (timesformer output) as input (my base model is [Neleac/timesformer-gpt2-video-captioning](https://huggingface.co/Neleac/timesformer-gpt2-video-captioning)), but there's no such flags in scripts or node input in graph. So GPT2 coverting as separate model without link to timesformer output.
So I was wondering if there are any plans to implement this option. I've tried manually manipulating the graph and script to no avail.
### Describe scenario use case
Usage of Encoder-Decoder (such as SpeechEncoderDecoderModel or VisionEncoderDecoderModel from HF)
Contributor guide
Research direction
Start by reading onnxruntime/python/tools/transformers/convert_generation.py and locating the GPT2_BeamSearch Node definition and graph inputs it creates. Compare the current GPT-2 beam-search path with the encoder-decoder scenario described in the issue, then identify the relevant tests or model-conversion checks. Done means the conversion script accepts encoder_hidden_states and the generated graph exposes and uses it for the requested model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100