huggingface / huggingface/transformers

[WIP] New Model Add FastPitch 1.1

Open
#16,349 4 comments 3 reactions 0 assignees View on GitHub
New model
Dominant language
Python
Stars
166k
Forks
34.6k
Avg merge
3d 9h
Merged PRs (30d)
281

Description

# 🌟 New model addition

## Model description

**What type of model is Fast Pitch 1.1?**
It is a Mel spectrogram generator (part of a speech to text model engine) that mainly comprises of two Feed Forward Transformer stacks.
Fast Pitch is used to transform text to a spectrogram **to be used in wave form generation in speech synthesis**.
Fast Pitch 1.1 include's multispeaker embeddings.
**What is the novel feature of the model making it different from other spectrogram generators?**
A fully-parallel text-to-speech model
based on Fast Speech, conditioned on fundamental frequency
contours. The model predicts pitch contours during inference.
By altering these predictions, the generated speech can be
more expressive, better match the semantic of the utterance,
and in the end more engaging to the listener.
Uniformly increasing or decreasing pitch with Fast Pitch generates speech
that resembles the voluntary modulation of voice.

Fast Pitch is meant to be used with a *neural vocoder* like Wave Net, or Wave Glow.

Text (Feature Extraction) → **Audio Synthesis (**spectrogram)**→ Waveform Synthesis (wavform)**

From the [paper](https://arxiv.org/pdf/2006.06873.pdf)
abstract:

> We present FastPitch, a fully-parallel text-to-speech model
> based on FastSpeech, conditioned on fundamental frequency
> contours. The model predicts pitch contours during inference.
> By altering these predictions, the generated speech can be
> more expressive, better match the semantic of the utterance,
> and in the end more engaging to the listener. Uniformly increasing or decreasing pitch with FastPitch generates speech
> that resembles the voluntary modulation of voice. Conditioning on frequency contours improves the overall quality of
> synthesized speech, making it comparable to state-of-the-art.
> It does not introduce an overhead, and FastPitch retains the
> favorable, fully-parallel Transformer architecture, with over
> 900× real-time factor for mel-spectrogram synthesis of a typical utterance.

## Open source status
**Samples**
https://fastpitch.github.io/
**My Own Generated Samples**
https://voca.ro/1eYmqidRhGi6
**Pro's of the model:**
It plays a part in a high MOS score. Compariable to Tacotron2, without the high cost of inference.

High performance and High Quality will be useful to provide voice or soul to digital assistants or metaverse assistants.

Training isn't sophisticated, unlike FastPitch 1.0 this model does not required durations or alignments to be generated from Tacotron2 or Montreal Forced Aligner.

**Con's of the model:**
It isn't in the Hugging Face repository to be easily adapted to products use cases. =)

* [x] the model implementation is available:
It is availible here.
https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/FastPitch
* [x] the model weights are available:
Using Automatic Mixed Percision FP-1.1
https://catalog.ngc.nvidia.com/orgs/nvidia/models/fastpitch_pyt_amp_ckpt_v1_1/files?version=21.12.0
* [x] who are the authors: @alancucki
*I am sorry if I missed anyone.*

cc @anton-l @patrickvonplaten
will assign to whom will be availible once the draft is complete.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the FastPitch implementation in NVIDIA/DeepLearningExamples/PyTorch/SpeechSynthesis/FastPitch, the linked model weights, and the referenced paper. The issue does not identify Transformers files, entry points, or tests; completion would require defining and implementing the FastPitch 1.1 model addition and verifying it in this repository.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
audio-video-rtc, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.