element-hq / element-hq/element-call

Transcription/Closed Captions

Open
#628 0 comments 2 reactions 0 assignees View on GitHub
A11y O-Occasional T-Enhancement X-Needs-Design X-Needs-Product
Dominant language
TypeScript
Stars
996
Forks
213
Avg merge
1d 8h
Merged PRs (30d)
55

Description

### Your use case

#### What would you like to do?
I would like to be able to have an option to enable closed captions (transcription) within an element call. This is comparable to Google Meet's closed captioning feature.

#### Why would you like to do it?
Having closed captions would provide element call with a major advantage compared to alternative services. Additionally, closed captioning is a needed accessibility tool which is able to help people who are deaf or hard of hearing and others with auditory processing issues. Not to mention, it is a great feature to have for people who want a text version of what is being spoken about in the call to better understand what is being said. There are numerous benefits to a feature like this.

#### How would you like to achieve it?
I believe this can be achieved securely with local transcribing options. This would solve any concerns with audio data being sent to a centralized place or being sent anywhere at all. Instead, each client with closed captions enabled would transcribe the audio they receive on the client side. I believe this can be done with tools such as vosk (https://github.com/ccoreilly/vosk-browser), coqui-STT (https://github.com/coqui-ai/STT though I'm not sure it is possible for this to work locally in browser), Web Speech API (https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API, https://www.google.com/intl/en/chrome/demos/speech.html, though this appears to only work in Chrome) and other alternatives. This would involve taking audio data being sent to the user with closed captions enabled, transcribing it (within the browser), and then outputting the transcription result so it is visible to the user with the feature enabled.

### Have you considered any alternatives?

The alternative would be to just not have this feature. However, I really think this would be a useful feature to have so please take this into consideration.

### Additional context

I'd really like to see this become a feature of element call. Feel free to let me know if you have any questions, I'll do my best to answer them.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.