huggingface / huggingface/transformers
Add FastSpeech2
- Dominant language
- Python
- Stars
- 166k
- Forks
- 34.6k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 281
Description
# 🌟 New model addition
## Model description
FastSpeech2 is a TTS model that outputs mel-spectrograms given some input text. From the [paper](https://arxiv.org/abs/2006.04558) abstract:
> Non-autoregressive text to speech (TTS) models such as FastSpeech can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at [this https URL](https://speechresearch.github.io/fastspeech2/).
## Open source status
* [x] the model implementation is available
* [x] the model weights are available
* [x] who are the authors: @RayeRen
The authors have not open-sourced their code implementation. However, the first author replied to an email inquiry and pointed me to the official implementation of DiffSinger, which includes FastSpeech2 code. This is likely the closest original implementation we can access.
* [DiffSinger](https://github.com/MoonInTheRiver/DiffSinger/tree/master/modules/fastspeech)
LJ Speech model weights are available [here](https://drive.google.com/file/d/1Zp45YjKkkv5vQSA7woHIqEggfyLqQdqs/view).
Other notable unofficial implementations include:
* [ming024](https://github.com/ming024/FastSpeech2)
* [ESPmet](https://espnet.github.io/espnet/_modules/espnet2/tts/fastspeech2/fastspeech2.html)
* [TensorFlowTTS](https://github.com/TensorSpeech/TensorFlowTTS/blob/master/tensorflow_tts/models/fastspeech2.py)
## Additional Context
This issue is a revisiting of https://github.com/huggingface/transformers/pull/11135.
cc @anton-l @patrickvonplaten
Contributor guide
Research direction
Start by reviewing the earlier Transformers pull request #11135 and the referenced DiffSinger modules/fastspeech implementation, then compare the ESPnet and TensorFlowTTS FastSpeech2 entry points. Done means FastSpeech2 is integrated into Transformers with the referenced LJ Speech weights and the model behavior is covered by the project’s validation checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100