google / google/lyra

Lyra v2 TF/JAX precursor model files? (trying to convert models to ONNX format)

Open
#99 10 comments 1 reaction 0 assignees View on GitHub
question
Dominant language
C++
Stars
4k
Forks
368
PR merge metrics
No merged PRs in 30d

Description

I'm trying to get Lyra v2 working in the browser via [tfjs-tflite](https://github.com/tensorflow/tfjs/tree/master/tfjs-tflite), and also [ORT Web](https://github.com/microsoft/onnxruntime/tree/main/js/web). On the ONNX side of things, I'm blocked by a conversion error and the [tf2onnx](https://github.com/onnx/tensorflow-onnx) team is unsure whether the tflite files are completely "valid", so they've asked that I look into whether there were some bugs in the conversion process, particularly with respect to the "NoOp" sub-graph of the encoder model:

https://github.com/onnx/tensorflow-onnx/issues/2055#issuecomment-1272512746

That thread shows netron.app images of what I'm referring to. There are `TFL_ASSIGN_VARIABLE` nodes which don't have any outputs, which is apparently not expected/valid.

Since the TF/JAX/(?) precursor to the tflite model isn't available in this repo, I'm not able to investigate further, so I was wondering if those original models could be published, or whether anyone on the Lyra team has any thoughts/expertise here?

If the original model files can't be published, I wonder if it would be possible for the team to convert the original models to ONNX, and then publish those ONNX files in this repo? Maybe there's something about the TF/JAX-->TFLite-->ONNX pathway that is causing the problem, and that wouldn't happen if we just went from TF/JAX straight to ONNX.

The script for the conversion, assuming that the original model is a SavedModel, would be:
```py
# 1. install tf2onnx:
pip install git+https://github.com/onnx/tensorflow-onnx
# 2. convert (assumes SavedModel folder is called 'soundstream_encoder' in current directory):
python -m tf2onnx.convert --opset 16 --saved-model soundstream_encoder --output soundstream_encoder.onnx --verbose --debug
# 3. repeat the above command for the other two models
```
I've tested with TensorFlow version 2.9.2. Note that versions 2.8 and 2.10 didn't work for me due to [this error](https://github.com/onnx/tensorflow-onnx/issues/2052) and another error which I can't remember.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.