google / google/lyra

In what configuration is the Soundstream in Lyra V2 trained?

Open
#102 9 comments 2 reactions 0 assignees View on GitHub
question
Dominant language
C++
Stars
4k
Forks
368
PR merge metrics
No merged PRs in 30d

Description

Referring to the original Soundstream article, Soundstream should be trained on 24kHz data. I would like to know what sample rate wavs these models released in lyraV2 (soundstream_encoder.tflite; quantizer.tflite; lyragan.tflite) were trained on. **Can these models also support processing 24kHz wavs?** Could these models be used on 24kHz wavs to do some interesting experiments similar to another Google work AudioLM.

I found that the existing models seem to be processing 16kHz wavs. However, I found in 48 line in [lyra_encoder.h](https://github.com/google/lyra/blob/main/lyra_encoder.h) the supported sample rates are not only 16000, but also 8000, 32000, and 48000.
This makes me confused. Different sample rate means that the fixed 320 samples vary in the different time span. I'm not quite sure if this fixed soundstream_encoder can directly handle data of different sample rates. Because given 46 4bit quantizers, the encoded data is not the supported bit rates (9.2kbps) mentioned in the API doc.
Actually, I use the three released models to encode, quantize and decode a 16Khz and a 24Khz wav with the same content, the two decode waves sound like the same. Due to the limitation of the num of test examples, I am not sure about the recovery quality. Can anyone explain this? Much thanks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.