JacobLinCool / JacobLinCool/Intervox

Add interpretation output modes: streaming and turn-taking

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Background

Intervox currently optimizes for low-latency streaming interpretation: translated speech is produced as soon as possible and can overlap with the original input. That is useful for fast interactive conversations, but it is not the only interpretation style users expect.

A second mode should support turn-taking/consecutive-style interpretation, where the user speaks a segment, Intervox waits for a pause or segment boundary, then plays the translated speech without overlapping the original speaker.

## Proposed modes

### Streaming interpretation

- Current direction: translate and output as soon as possible.
- Can overlap with ongoing input speech.
- Optimized for low latency and conversational pace.
- Best for short utterances and users who accept some overlap.

### Turn-taking interpretation

- Buffer input until a pause, sentence, or utterance boundary.
- Play translated speech after the source segment ends.
- Avoid overlapping original speech and translated speech.
- Optimized for clarity, formal meetings, and situations where the listener needs a cleaner translated channel.
- Higher latency is acceptable and expected.

## Implementation direction

- Add an explicit interpretation output strategy setting, likely under Translation settings.
- Model the strategy in shared config/state rather than front-end-only UI state.
- Keep the current streaming path as one strategy.
- Add a turn-taking state machine around capture/transcript/translation/playback boundaries.
- Use VAD and/or realtime transcript segment events to decide when a source segment is complete enough to translate/play.
- In turn-taking mode, queue translated audio and only write it to the virtual mic when the source side is not actively speaking.
- Define behavior for users who continue speaking while translation playback is queued or active.
- Ensure captions communicate the current state: listening, translating, playing translation, waiting for pause, etc.
- Avoid ad hoc fallback behavior where the app sometimes overlaps anyway without surfacing that state.

## Product questions

- Should turn-taking mode pause/suppress virtual mic output while the user is speaking, or should it queue only translated audio while still allowing original pass-through depending on mix settings?
- Should original voice mix be disabled or constrained in turn-taking mode?
- What pause threshold should define an utterance boundary by default?
- Do we need a push-to-interpret / manual segment boundary option later?

## Acceptance criteria

- Users can choose between Streaming and Turn-taking interpretation modes.
- Streaming mode preserves current low-latency behavior.
- Turn-taking mode does not intentionally overlap translated speech with active source speech.
- Turn-taking mode buffers/segments input and plays translated speech after a source pause or segment boundary.
- Translation playback queues are reset correctly when switching modes, stopping interpretation, or encountering an error.
- Captions/status UI reflects the active strategy and current turn-taking state.
- Add focused tests for strategy selection, state transitions, queue reset, and duplicate/overlap prevention where practical.
- Manual acceptance notes cover a realistic meeting flow for both modes.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the shared config/state and the capture, transcript, translation, and playback boundaries described in the issue. Review how VAD or realtime transcript segment events can define source completion, then map the state transitions and queue-reset cases before choosing behavior for unresolved product questions. Done includes focused strategy, transition, queue, and overlap tests plus manual meeting-flow notes for both modes.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, rust
Domain
audio-video-rtc, desktop
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.