element-hq / element-hq/element-meta

Automatic Transcription of Voice Messages

Open
#650 0 comments 9 reactions 0 assignees View on GitHub
A-Voice Messages O-Occasional S-Minor T-Enhancement X-Needs-Product
Dominant language
No language data
Stars
112
Forks
25
Avg merge
6h 6m
Merged PRs (30d)
4

Description

**Is your feature request related to a problem? Please describe.**
Sometimes when I receive a voice message, I am not in an environment where I am able to listen to the message. Additionally, I may be listening to something already and not want to add an additional stream of audio to be able to communicate with someone. Finally, I may be deaf and not able to hear audio at all.

**Describe the solution you'd like**
In the above cases, it would be helped to have the ability to automatically transcribe a voice message into text. Ideally this would be done on-device without needing to contact a cloud provider (or pay a provider a fee to do so).

Several options for on-device transcription for human speech exist today:

| Project | Description | Performance | License |
| --- | --- | --- | --- |
| [Coqui STT](https://github.com/coqui-ai/STT) | Already includes libraries for Java, JS, WASM (experimental) and C. Also they have [a Matrix Space](https://matrix.to/#/+coqui:matrix.org). | Advertised as production-ready.
[Try it out in Google Colab](https://colab.research.google.com/github/coqui-ai/STT/blob/main/notebooks/train_your_first_coqui_STT_model.ipynb). | MPL 2.0 |
| OpenAI Whisper | Features English-only and multi-lingual models. No dedicated library (yet?), just PyTorch models (which can be run anywhere). | The initial release includes a range of different models with differing performance, from Tiny (English-only, 72MB) to Large (Multi-lingual, 3GB). Likely the better performing one.
[Try it out in Google Colab](https://colab.research.google.com/github/openai/whisper/blob/master/notebooks/LibriSpeech.ipynb). | MIT |

Ratio of real-time performance would need to be tested for each model on a range of devices. Longer voice messages will take longer to transcribe.

Running on the client versus the homeserver is preferred, especially for voice messages sent in end-to-end encrypted rooms.

**Describe alternatives you've considered**
Using a third-party service for voice transcription would be both a privacy leak and not monetarily sustainable.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.