magenta / magenta/ddsp

Here's an example to transfer timbre in browser [Docs]

Open
#482 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
3.4k
Forks
384
PR merge metrics
No merged PRs in 30d

Description

Pulled my hair out over this one, so leaving it here for anyone else who needs it:

```
{
const Magenta = (await import('@magenta/music'))
const { DDSP, SPICE } = Magenta
const spice = new SPICE()
await spice.initialize()

// const audioUrl = 'https://s3-us-west-2.amazonaws.com/s.cdpn.io/123941/Yodel_Sound_Effect.mp3'; // yodel

const context = new AudioContext();

window.fetch(audioUrl)
.then(async response => {
const arrayBuffer = await response.arrayBuffer()
return arrayBuffer
})
.then(async arrayBuffer => {
const audioBuffer = await context.decodeAudioData(arrayBuffer)
return audioBuffer
})
.then(async audioBuffer => {
const audioFeatures = await spice.getAudioFeatures(audioBuffer)
const checkpointUrl = 'https://storage.googleapis.com/magentadata/js/checkpoints/ddsp/violin'
const ddsp = new DDSP(checkpointUrl)
await ddsp.initialize()
const synthesizedBuffer = await ddsp.synthesize(audioFeatures)

// Copied from ~/node_modules/@magenta/music/esm/ddsp/buffer_utils.js, which is
// called by the addReverb method in node_modules/@magenta/music/esm/ddsp/add_reverb.js,
// which is called by the .synthesize method in node_modules/@magenta/music/esm/ddsp/model.js
const arrayBufferToAudioBuffer = (audioCtx, arrayBuffer, sampleRate) => {
const newBuffer = audioCtx.createBuffer(1, arrayBuffer.length, sampleRate);
newBuffer.copyToChannel(arrayBuffer, 0);
return newBuffer;
};
console.log('synthesizedBuffer:', synthesizedBuffer)

const synthesizedAudioBuffer = arrayBufferToAudioBuffer(context, synthesizedBuffer, 48000)

function play(myBuffer) {
const source = context.createBufferSource();
source.buffer = myBuffer;
source.connect(context.destination);
source.start();
}

play(synthesizedAudioBuffer)
});
}}>


```

Checkpoint info (for violin, flute, etc.) can be found here:

https://github.com/magenta/magenta-js/tree/master/music/checkpoints#table

In my experience, the checkpoint downloader script didn't really work. It also appears there are 2 generations of DDSP models: one that powers this site here...

https://magenta.github.io/magenta-js/music/demos/ddsp_tone_transfer.html

...and then one that powers the Google Colab they link to on the front page of the DDSP README:

https://colab.research.google.com/github/magenta/ddsp/blob/main/ddsp/colab/demos/timbre_transfer.ipynb

I've only been able to access the gen 1 checkpoints so far. Haven't really tried with gen 2, but my plan for that one is to run the Google Colab, and manually download the checkpoint files from disk on the left, and then host them at S3. That will also need to host the `.pkl` and `.gin` files too, AFAIK.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the browser example, the linked checkpoint table, and the DDSP demo and Colab links first. Done means the timbre-transfer example and checkpoint guidance are documented clearly, including the noted downloader and model-generation caveats.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, python
Domain
audio-video-rtc, documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.