huggingface / huggingface/transformers
[WIP] New Model Add FastPitch 1.1
- Dominant language
- Python
- Stars
- 166k
- Forks
- 34.6k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 281
Description
# 🌟 New model addition
## Model description
**What type of model is Fast Pitch 1.1?**
It is a Mel spectrogram generator (part of a speech to text model engine) that mainly comprises of two Feed Forward Transformer stacks.
Fast Pitch is used to transform text to a spectrogram **to be used in wave form generation in speech synthesis**.
Fast Pitch 1.1 include's multispeaker embeddings.
**What is the novel feature of the model making it different from other spectrogram generators?**
A fully-parallel text-to-speech model
based on Fast Speech, conditioned on fundamental frequency
contours. The model predicts pitch contours during inference.
By altering these predictions, the generated speech can be
more expressive, better match the semantic of the utterance,
and in the end more engaging to the listener.
Uniformly increasing or decreasing pitch with Fast Pitch generates speech
that resembles the voluntary modulation of voice.
Fast Pitch is meant to be used with a *neural vocoder* like Wave Net, or Wave Glow.
Text (Feature Extraction) → **Audio Synthesis (**spectrogram)**→ Waveform Synthesis (wavform)**
From the [paper](https://arxiv.org/pdf/2006.06873.pdf)
abstract:
> We present FastPitch, a fully-parallel text-to-speech model
> based on FastSpeech, conditioned on fundamental frequency
> contours. The model predicts pitch contours during inference.
> By altering these predictions, the generated speech can be
> more expressive, better match the semantic of the utterance,
> and in the end more engaging to the listener. Uniformly increasing or decreasing pitch with FastPitch generates speech
> that resembles the voluntary modulation of voice. Conditioning on frequency contours improves the overall quality of
> synthesized speech, making it comparable to state-of-the-art.
> It does not introduce an overhead, and FastPitch retains the
> favorable, fully-parallel Transformer architecture, with over
> 900× real-time factor for mel-spectrogram synthesis of a typical utterance.
## Open source status
**Samples**
https://fastpitch.github.io/
**My Own Generated Samples**
https://voca.ro/1eYmqidRhGi6
**Pro's of the model:**
It plays a part in a high MOS score. Compariable to Tacotron2, without the high cost of inference.
High performance and High Quality will be useful to provide voice or soul to digital assistants or metaverse assistants.
Training isn't sophisticated, unlike FastPitch 1.0 this model does not required durations or alignments to be generated from Tacotron2 or Montreal Forced Aligner.
**Con's of the model:**
It isn't in the Hugging Face repository to be easily adapted to products use cases. =)
* [x] the model implementation is available:
It is availible here.
https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/FastPitch
* [x] the model weights are available:
Using Automatic Mixed Percision FP-1.1
https://catalog.ngc.nvidia.com/orgs/nvidia/models/fastpitch_pyt_amp_ckpt_v1_1/files?version=21.12.0
* [x] who are the authors: @alancucki
*I am sorry if I missed anyone.*
cc @anton-l @patrickvonplaten
will assign to whom will be availible once the draft is complete.
Contributor guide
Research direction
Start by reviewing the FastPitch implementation in NVIDIA/DeepLearningExamples/PyTorch/SpeechSynthesis/FastPitch, the linked model weights, and the referenced paper. The issue does not identify Transformers files, entry points, or tests; completion would require defining and implementing the FastPitch 1.1 model addition and verifying it in this repository.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- audio-video-rtc, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100