huggingface / huggingface/sharp-transformers

It is hoped that feature processing of the whisper model can be supported in the future:)

Open
#2 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
C#
Stars
200
Forks
17
PR merge metrics
No merged PRs in 30d

Description

In the whisper model, it is exported to three onnx, namely encoder_model.onnx, decoder_model.onnx, decoder_with_past_model.onnx. The input dimension of encoder_model.onnx is fbank feature data of [1, 80, 300] dimension, but the data to extract fbank features is written in huggingface in python, so I hope that all about whisper can be used in future versions. Here are some personal information Links that I found useful :)

Looking forward to sharp-transformers getting better and better

Fbank feature extraction
https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/feature_extraction_whisper.py#L84
https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/feature_extraction_whisper.py#L250
https://github.com/huggingface/transformers/blob/main/src/transformers/audio_utils.py#L119

WhisperToken
https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/tokenization_whisper.py#L52
https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/tokenization_whisper.py#L744

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.