DDSP components in TorchAudio

Differential Digital Signal Processing (DDSP) is a technique proposed by the Google Magenta team. [repo, papers].

It provides powerful generative methods that take advantage of the structure of sound based on Fourier analysis.
The paper has over 200 citations and is being applied to different tasks like speech synthesis and source separation.

TorchAudio is looking to support DDSP-based modeling methods by adding basic components.
We would like to hear and get help from the community on this work stream.

Initial Goal

To support DDSP-based generative modeling.

New Components / APIs

1. Harmonic Synthesizer / Additive Synthesis
   • Differentiable harmonic synthesizer
     • Oscillator bank #2848
     • Harmonic overtones #2863
   • ADSR envelope #2859
2. Subtractive Synthesis
   • Filter design functions
     • windowed-sinc filter #2875
     • frequency method #2879
   • filter function #2928

Components for DDSP Modeling from [1]

1. Multi-scale mel spectrogram loss
   Used to train models. Compare the melspectrogram of the original waveform and synthesized waveform at multiple scales. [reference impl]
2. exp_sigmoid
   Modified sigmoid function used to ensure that model output is in the value range expected by synthesizers [reference impl]
3. [Nice to have] Dataset
   • NSynth
   • Solo Violin
4. [Stretch] DDSP Model from [1]
   • AutoEncoder model for Timbre transfer
5. [Stretch] Training Script and pre-trained model
   Script to train models for
   • LJSpeech
   • Solo violin

Existing Components / Workstream

• convolve and fftconvolve will be the basis for filter application.
• RIR simulation can be used for reverb.

Existing Tutorials

• Filter design
• Subtractive synthesis
• Oscillator and ADSR envelope
• Additive synthesis

References

1. [2001.04643] DDSP: Differentiable Digital Signal Processing

and some random papers moto found interesting

1. [2010.15084] Speech Synthesis and Control Using Differentiable DSP
2. [2202.00200] Differentiable Digital Signal Processing Mixture Model for Synthesis Parameter Extraction from Mixture of Harmonic Sounds
3. [2210.14476] Sinusoidal Frequency Estimation by Gradient Descent

Start by reviewing the DDSP paper, the Magenta reference implementations, and TorchAudio's existing convolve, fftconvolve, RIR simulation, and synthesis tutorials. The issue lists several completed components alongside broad, unchecked modeling, dataset, and training goals, so a contributor needs a maintainer to define one concrete scope and acceptance criteria before work can be considered done.