Google's Trillson Audio Classification

Model description

The TRILLsson models are described in the publication TRILLsson: Distilling Universal Paralingistic Speech Representations. From audio, they generate generally-useful paralinguistic speech representations (paralinguistics are aspects of speech other than text, such as emotion, language identification, synthetic or real, etc). These representations are smaller, faster, and publicly available versions of the state-of-the-art CAP12 embeddings, which are described in Universal Paralinguistic Speech Representations Using Self-Supervised Conformers (ICASSP 2022).

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

Google recently has done some very nice work on better audio / speech representations and distilled audio / speech representations. See:

• https://arxiv.org/abs/2110.04621
• https://arxiv.org/abs/2203.00236

Some of the distilled models are open-sourced and could be made more available via an integration to HuggingFace's Transformer library.

E.g. the following notebook shows how the weights can be loaded and run with publicly accessible model code:
https://colab.research.google.com/drive/1-D6pyxFyquIO8pss_lngL_mncHa3kAAT?usp=sharing

The relevent models to add are:

• https://tfhub.dev/google/nonsemantic-speech-benchmark/trillsson3/1 and
• https://tfhub.dev/google/nonsemantic-speech-benchmark/trillsson2/1

and the relevant code is publicly available: https://github.com/google-research/google-research/tree/master/non_semantic_speech_benchmark

The google colab shows exacty how the model can be run and debugged in TF.

Start with the linked Google Colab notebook and the google-research non_semantic_speech_benchmark code to understand how the trillsson2 and trillsson3 weights are loaded and run. Compare the implementation with the Transformers integration points, then verify that both TensorFlow Hub models are available through the library with equivalent audio classification behavior.