huggingface / huggingface/transformers

Support for Monotonic Mulithead Attention based Simultaneous Speech-to-text Translation

Open
#15,491 1 comment 0 reactions 0 assignees View on GitHub
New model
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 9h
Merged PRs (30d)
281

Description

# 🌟 New model addition
Simultaneous Speech-to-text Translation using Monotonic Multihead Attention(MMA).
I am wondering if anybody is working on implementing this model for now.
However, I am worried that if this model is going to be supported by Hugging Face systems, since inference works in a particular way using frameworks like [SimulEval](https://github.com/facebookresearch/SimulEval) to simulate streaming input which may not be compatible with current Hugging Face's inference system?

## Model description
[MMA(Ma et al., 2019)](https://arxiv.org/abs/1909.12406) has been used to handle streaming text/speech inputs mostly for translation, where MMA extends the monotonic attention mechanism to multihead.

## Open source status
* [x] the model implementation is available: [Fairseq Implementation is available here](https://github.com/pytorch/fairseq/blob/fcca32258c8e8bcc9f9890bf4714fa2f96b6b3e1/examples/simultaneous_translation/models/convtransformer_simul_trans.py#L29~#L63)
* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username) : Xutai Ma(@xutaima), Juan Pino, James Cross, Liezl Puzon, Jiatao Gu

Inference framework : [Facebook Research SimulEval](https://github.com/facebookresearch/SimulEval)

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked Fairseq implementation in examples/simultaneous_translation/models/convtransformer_simul_trans.py and the SimulEval inference framework. Clarify how streaming inference should fit Hugging Face systems, whether model weights will be available, and what acceptance criteria define support; the issue currently leaves these decisions unresolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
audio-video-rtc, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.