Here's an example to transfer timbre in browser [Docs]
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 384
- PR merge metrics
- No merged PRs in 30d
Description
Pulled my hair out over this one, so leaving it here for anyone else who needs it:
```
{
const Magenta = (await import('@magenta/music'))
const { DDSP, SPICE } = Magenta
const spice = new SPICE()
await spice.initialize()
// const audioUrl = 'https://s3-us-west-2.amazonaws.com/s.cdpn.io/123941/Yodel_Sound_Effect.mp3'; // yodel
const context = new AudioContext();
window.fetch(audioUrl)
.then(async response => {
const arrayBuffer = await response.arrayBuffer()
return arrayBuffer
})
.then(async arrayBuffer => {
const audioBuffer = await context.decodeAudioData(arrayBuffer)
return audioBuffer
})
.then(async audioBuffer => {
const audioFeatures = await spice.getAudioFeatures(audioBuffer)
const checkpointUrl = 'https://storage.googleapis.com/magentadata/js/checkpoints/ddsp/violin'
const ddsp = new DDSP(checkpointUrl)
await ddsp.initialize()
const synthesizedBuffer = await ddsp.synthesize(audioFeatures)
// Copied from ~/node_modules/@magenta/music/esm/ddsp/buffer_utils.js, which is
// called by the addReverb method in node_modules/@magenta/music/esm/ddsp/add_reverb.js,
// which is called by the .synthesize method in node_modules/@magenta/music/esm/ddsp/model.js
const arrayBufferToAudioBuffer = (audioCtx, arrayBuffer, sampleRate) => {
const newBuffer = audioCtx.createBuffer(1, arrayBuffer.length, sampleRate);
newBuffer.copyToChannel(arrayBuffer, 0);
return newBuffer;
};
console.log('synthesizedBuffer:', synthesizedBuffer)
const synthesizedAudioBuffer = arrayBufferToAudioBuffer(context, synthesizedBuffer, 48000)
function play(myBuffer) {
const source = context.createBufferSource();
source.buffer = myBuffer;
source.connect(context.destination);
source.start();
}
play(synthesizedAudioBuffer)
});
}}>
```
Checkpoint info (for violin, flute, etc.) can be found here:
https://github.com/magenta/magenta-js/tree/master/music/checkpoints#table
In my experience, the checkpoint downloader script didn't really work. It also appears there are 2 generations of DDSP models: one that powers this site here...
https://magenta.github.io/magenta-js/music/demos/ddsp_tone_transfer.html
...and then one that powers the Google Colab they link to on the front page of the DDSP README:
https://colab.research.google.com/github/magenta/ddsp/blob/main/ddsp/colab/demos/timbre_transfer.ipynb
I've only been able to access the gen 1 checkpoints so far. Haven't really tried with gen 2, but my plan for that one is to run the Google Colab, and manually download the checkpoint files from disk on the left, and then host them at S3. That will also need to host the `.pkl` and `.gin` files too, AFAIK.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the browser example, the linked checkpoint table, and the DDSP demo and Colab links first. Done means the timbre-transfer example and checkpoint guidance are documented clearly, including the noted downloader and model-generation caveats.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, python
- Domain
- audio-video-rtc, documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100