google / google/lyra

Correct tflite model usage/pipeline?

Open
#92 11 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
C++
Stars
4k
Forks
368
PR merge metrics
No merged PRs in 30d

Description

My current guess/understanding is:

1. Get 320 samples of a 16 khz audio file
2. Feed that into `soundstream_encoder.tflite` as `float32[1,320]` and get `float32[1,1,64]` data as output.
3. (???) Somehow quantize, transmit over network, and then dequantize using `quantizer.tflite`? It has an input shape of `int32[46,1,1]`, so I'm not sure what to do with that.
4. Feed the output of dequantization process (`float32[1,1,64]`) to `lyragan.tflite` to produce the reconstructed samples (`float32[1,320]`)

I'm not able to test this guess/understanding right now because I'm blocked on [some issues](https://huggingface.co/rocca/lyra-v2-soundstream), but I'm aiming to get a minimal open source web demo working using either [tfjs-tflite](https://github.com/tensorflow/tfjs/tree/master/tfjs-tflite), or [ONNX Runtime Web](https://github.com/microsoft/onnxruntime/tree/main/js/web), so if anyone could correct my understanding here, that would be great!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.