microsoft / microsoft/onnxruntime
[Feature Request] Add "output_hidden_states" option to convert_generation.py
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
I can successfuly use the [convert_generation.py](https://github.com/microsoft/onnxruntime/blob/0c6037b5abe571fc43a55ef7a9d2f846820fbe5d/onnxruntime/python/tools/transformers/convert_generation.py) script to create a T5 code generation model with beam search (my base model is [Salesforce/codet5-base-multi-sum](https://huggingface.co/Salesforce/codet5-base-multi-sum)). However, I'd also like to output the `encoder_hidden_states` output from the encoder subgraph to use in a downstream task. [Huggingface](https://huggingface.co/docs/transformers/internal/generation_utils#transformers.generation_utils.BeamSearchDecoderOnlyOutput) has an `output_hidden_states` flag to append this tensor to the output, but I don't see this option here. I noticed that the `output_sequence_scores` and `output_token_scores` options exist like they do in Hugginface, so I was wondering if there are any plans to implement this option. I've tried manually manipulating the graph to no avail.
### Describe scenario use case
Usage of `encoder_hidden_states` in downstream tasks.
Contributor guide
Research direction
Start with onnxruntime/python/tools/transformers/convert_generation.py and trace the existing output_sequence_scores and output_token_scores options. Compare those paths with the requested Hugging Face output_hidden_states behavior and determine how encoder_hidden_states should be exposed for beam-search generation. Done means the conversion script accepts the option and generated outputs include the encoder hidden states for downstream use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100