element-hq / element-hq/element-x-android
On-device voice message transcription via Whisper (working implementation and contribution RFC)
- Dominant language
- Kotlin
- Stars
- 2.4k
- Forks
- 624
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 149
Description
Voice message transcription has been requested across Element and Matrix clients for years (going back to the original Element Android repo). Voice messages are hard to skim in quiet environments and cannot be searched. An on-device transcription engine solves this without compromising end-to-end encryption or privacy.
I have built a working implementation in my fork and want to check a few technical details with the team before opening the pull requests.
## What is implemented
* **Core Engine (`features/voicetranscription` module):**
* Uses `sherpa-onnx` for local Whisper inference with a native bridge and serialized access.
* Audio decoding via `MediaCodec` to 16 kHz mono.
* Model manager for on-demand downloads. Sizes and content hashes (SHA-256 for LFS files, git-blob SHA-1 for plain files) are probed from the server before each download, so nothing is pinned in code and upstream model updates need no app change.
* An automatic RAM unload timer frees the native model 2 minutes after last use.
* In-memory transcript cache only (transcripts are never written to disk).
* Single-run execution: the engine holds one resident model, so concurrent runs and mid-run model switches are rejected and the UI disables the affected controls until a run finishes.
* **UI & Integration:**
* **Settings:** Master switch and model manager supporting the multilingual `tiny` (104 MB), `base` (161 MB) and `small` (375 MB) models. `tiny` is preselected by default, about 640 MB if all three are kept on disk. Whisper models are never bundled; they are downloaded on demand after the user enables the feature.
* **Timeline:** Transcripts render directly inside the audio message bubble, a transcribe button sits on the player row, with inline progress indicators. The spoken language is detected by Whisper itself, per message, without heuristics or configuration.
* **Privacy:** All processing happens locally on the device. Decrypted audio is written to a temporary file in the app-private cache only for the duration of a transcription run and is deleted immediately afterwards (same pattern as media playback).
## Questions before opening PRs
1. **APK Size / Native Dependencies:** The sherpa-onnx native libraries ship with every install. Measured on debug builds, the per-ABI APK grows by about 37 MB (163.6 to 200.5 MB for arm64-v8a), of which ~31 MB are four `.so` files (onnxruntime plus sherpa-onnx JNI/C APIs). Release builds should be somewhat smaller, but native code dominates. Is that trade-off acceptable for an opt-in feature, or would you prefer a different approach (e.g. downloading the runtime on demand as well, if the team considers that viable)?
2. **Model Hosting:** Models currently download from HuggingFace. Should these be mirrored on Element infrastructure instead?
3. **UI / Compound Design:** The layout follows familiar voice message STT patterns (similar to Telegram). I am happy to tweak spacing, components, or icons to align with Compound guidelines. I can provide screenshots or a short screen recording if needed.
4. **PR Breakdown:** The work is around 2,700 lines in total. I currently have it split into 2 branches:
* **PR 1 — Engine, model management (including hash verification) and the settings screen:**
[`feature/voicetranscription-engine`](https://github.com/moccassins/element-x-android/tree/feature/voicetranscription-engine)
* **PR 2 — Timeline integration (transcript bubble, transcribe button, timestamp row, in-chat model picker):**
[`feature/voicetranscription-timeline`](https://github.com/moccassins/element-x-android/tree/feature/voicetranscription-timeline)
Does this split work for review, or would you prefer a single PR or a different structure?
## Related Issues
* https://github.com/element-hq/element-android/issues/4244
* https://github.com/element-hq/element-android/issues/3857
* https://github.com/element-hq/element-android/issues/7483
Contributor guide
Research direction
Start by reviewing the features/voicetranscription module and the two proposed branches: feature/voicetranscription-engine and feature/voicetranscription-timeline. First seek team decisions on native dependency size, model hosting, Compound UI alignment, and PR structure; done means those review questions are resolved and the contribution plan is accepted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, kotlin
- Domain
- audio-video-rtc, mobile
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100