TEN-framework / TEN-framework/ten-framework
[FEATURE] Enable ASR translation outputs for ASR vendors that support real-time translation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11.1k
- Forks
- 1.4k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 22
Description
Background
Several ASR vendors (including Soniox and Speechmatics) offer real-time translation capabilities, but current ASR extensions don't expose translation outputs through their configuration parameters. Additionally, the standard ASR API in ten_ai_base lacks any specification for handling translation results. Supporting translation in ASR extensions would enable valuable use cases like live translation.
API
The standard ASR API in ten_ai_base currently provides the asr_result Data output. To support translation, we propose adding an optional asr_translation_result Data output with identical properties to asr_result. These two outputs operate independently—each maintains its own audio alignment via start_ms and duration_ms fields, without requiring synchronization between transcription and translation results.
Implementation
Decoupling asr_result from asr_translation_result significantly simplifies implementation across different vendor APIs, which handle translation differently:
-
Soniox ([docs](https://soniox.com/docs/stt/rt/real-time-translation)): Emits transcription tokens with timestamps, followed by corresponding translation tokens without timestamps. Translation tokens inherit timing from their associated transcription tokens. Both can be mapped to their respective outputs using the same timestamp values.
-
Speechmatics ([docs](https://docs.speechmatics.com/api-ref/realtime-transcription-websocket#addtranslation)): Provides separate message types (AddPartialTranscription, AddTranscription, AddPartialTranslation, AddTranslation), each with independent audio-aligned timestamps. These can be directly mapped one-to-one to
asr_resultandasr_translation_result.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the standard ASR API in ten_ai_base and inspect the existing asr_result Data output and the ASR extensions for Soniox and Speechmatics. Compare how each vendor represents transcription and translation timing. Done means an optional asr_translation_result output is specified with independent audio alignment and can represent the supported vendor translation results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, api, audio-video-rtc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100