microsoft / microsoft/onnxruntime

[Feature Request] Implement encoder_hidden_states as input in GPT2_BeamSearch Node

Open
#18,050 2 comments 0 reactions 0 assignees View on GitHub
feature request model:transformer
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

I try to use the [convert_generation.py](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/convert_generation.py) script to create a GPT2 code generation model with beam search with encoder_hidden_states (timesformer output) as input (my base model is [Neleac/timesformer-gpt2-video-captioning](https://huggingface.co/Neleac/timesformer-gpt2-video-captioning)), but there's no such flags in scripts or node input in graph. So GPT2 coverting as separate model without link to timesformer output.

So I was wondering if there are any plans to implement this option. I've tried manually manipulating the graph and script to no avail.

### Describe scenario use case

Usage of Encoder-Decoder (such as SpeechEncoderDecoderModel or VisionEncoderDecoderModel from HF)

Contributor guide

Open the contributing guide

Research direction

Start by reading onnxruntime/python/tools/transformers/convert_generation.py and locating the GPT2_BeamSearch Node definition and graph inputs it creates. Compare the current GPT-2 beam-search path with the encoder-decoder scenario described in the issue, then identify the relevant tests or model-conversion checks. Done means the conversion script accepts encoder_hidden_states and the generated graph exposes and uses it for the requested model.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.