magenta / magenta/ddsp

Question About Midi Autoencoder

Open
#511 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
384
PR merge metrics
No merged PRs in 30d

Description

I see that the Midi Autoencoder encodes all the F0 series, loudness series, harmonic distribution, and filtered noise down to a MIDI representation, then decodes it back to a F0 series and loudness series. I understand that the goal is to use the MIDI decoder at inference time when all you have is MIDI information and you need detailed F0/loudness series.

What I don't understand is how the MIDI Autoencoder is constrained to actually produce MIDI information in its latent space during training. I see that the encoder is forced to bottleneck its input information down to a representation that has the correct dimensionality for MIDI, and I see how the MIDI decoder transfers it back into the inputs that DDSP resynthesis requires. However, I don't understand how the model is constrained to actually produce MIDI information in the middle. As far as I understand, it could produce any latent representation in that dimensionality.

Let me know if I can clarify in any way.

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by locating the MIDI Autoencoder's encoder, decoder, and training path, then identify the loss or other constraint that is intended to shape the latent representation. Done means providing a clear explanation of how MIDI structure is enforced during training.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
audio-video-rtc, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.