In what configuration is the Soundstream in Lyra V2 trained?
- Dominant language
- C++
- Stars
- 4k
- Forks
- 368
- PR merge metrics
- No merged PRs in 30d
Description
Referring to the original Soundstream article, Soundstream should be trained on 24kHz data. I would like to know what sample rate wavs these models released in lyraV2 (soundstream_encoder.tflite; quantizer.tflite; lyragan.tflite) were trained on. **Can these models also support processing 24kHz wavs?** Could these models be used on 24kHz wavs to do some interesting experiments similar to another Google work AudioLM.
I found that the existing models seem to be processing 16kHz wavs. However, I found in 48 line in [lyra_encoder.h](https://github.com/google/lyra/blob/main/lyra_encoder.h) the supported sample rates are not only 16000, but also 8000, 32000, and 48000.
This makes me confused. Different sample rate means that the fixed 320 samples vary in the different time span. I'm not quite sure if this fixed soundstream_encoder can directly handle data of different sample rates. Because given 46 4bit quantizers, the encoded data is not the supported bit rates (9.2kbps) mentioned in the API doc.
Actually, I use the three released models to encode, quantize and decode a 16Khz and a 24Khz wav with the same content, the two decode waves sound like the same. Due to the limitation of the num of test examples, I am not sure about the recovery quality. Can anyone explain this? Much thanks.
Contributor guide
Assessment
This issue has not been assessed yet.