microsoft / microsoft/onnxruntime

[Feature Request] Add "output_hidden_states" option to convert_generation.py

Open
#13,685 3 comments 3 reactions 0 assignees View on GitHub
feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

I can successfuly use the [convert_generation.py](https://github.com/microsoft/onnxruntime/blob/0c6037b5abe571fc43a55ef7a9d2f846820fbe5d/onnxruntime/python/tools/transformers/convert_generation.py) script to create a T5 code generation model with beam search (my base model is [Salesforce/codet5-base-multi-sum](https://huggingface.co/Salesforce/codet5-base-multi-sum)). However, I'd also like to output the `encoder_hidden_states` output from the encoder subgraph to use in a downstream task. [Huggingface](https://huggingface.co/docs/transformers/internal/generation_utils#transformers.generation_utils.BeamSearchDecoderOnlyOutput) has an `output_hidden_states` flag to append this tensor to the output, but I don't see this option here. I noticed that the `output_sequence_scores` and `output_token_scores` options exist like they do in Hugginface, so I was wondering if there are any plans to implement this option. I've tried manually manipulating the graph to no avail.

### Describe scenario use case

Usage of `encoder_hidden_states` in downstream tasks.

Contributor guide

Open the contributing guide

Research direction

Start with onnxruntime/python/tools/transformers/convert_generation.py and trace the existing output_sequence_scores and output_token_scores options. Compare those paths with the requested Hugging Face output_hidden_states behavior and determine how encoder_hidden_states should be exposed for beam-search generation. Done means the conversion script accepts the option and generated outputs include the encoder hidden states for downstream use.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.