android / android/media-samples

AccessibilityService + TTS/STT: How to keep YouTube playing, silence STT beeps, and make TTS↔STT handoff feel natural?

Open
#116 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Kotlin
Stars
1.4k
Forks
781
PR merge metrics
No merged PRs in 30d

Description

Environment

Android 14/15 (including Samsung devices)
An AccessibilityService detects order popups from a dispatch app overlay
Order details are spoken with TextToSpeech, and SpeechRecognizer listens for voice commands such as “confirm” / “cancel”
Drivers often keep YouTube playing while using the app

Goals (all three at once)

Do not interrupt YouTube (or other media) — no pause / stop, and avoid 1-second ducking loops
No STT start/stop/error beeps (the system “ding / ting” sound)
TTS and STT should work smoothly together — STT should become ready right after (or overlapping with) TTS so voice confirm feels natural

What we’ve tried / side effects

Audio focus

TTS with AUDIOFOCUS_GAIN_TRANSIENT_MAY_DUCK → YouTube sometimes ducks or pauses
Requesting audio focus before/during STT → LOSS_TRANSIENT kills STT, or YouTube pauses
Requesting no focus keeps YouTube alive, but during media playback STT onReady is often late and ERROR_NO_MATCH (7) becomes common

Beep suppression

RecognizerIntent extras such as android.speech.extra.BEEP_SOUND=false, sound_start_beep=0 → ignored on some devices/engines
Temporarily muting only STREAM_NOTIFICATION / STREAM_SYSTEM → reduces beeps, but mute/restore ping-pong can still affect YouTube or system volume
Muting STREAM_MUSIC is unacceptable because it kills YouTube

TTS ↔ STT timing

Blocking startListening during TTS → STT resume after TTS is often delayed (seconds to tens of seconds)
Opening TTS and STT at the same time → mic contention with Google STT/SODA: TTS delay, or “ghost listen” (isListening=true but no onReady / no RMS)
Lost or delayed TTS onDone can leave the STT window never opened

Media + NO_MATCH

Immediately calling startListening again on every ERROR_NO_MATCH while YouTube plays causes ~1s pause/duck ping-pong
Backing off too much (e.g. 4–8s) makes users miss the confirm window

Questions
For an Accessibility-based field voice UI, is there a recommended pattern that satisfies all of the following?

STT that coexists with YouTube / isMusicActive media (do not disable STT just because a media app is in the foreground)
Fully suppress SpeechRecognizer/system beeps without volume hacks — or with the safest minimal hack
Start STT listening within ~1 second after TTS speak, ideally with an overlapping handoff that doesn’t fight for the mic/focus

In particular:

Which AudioAttributes / usage / contentType should TTS and STT each use?
Should we request audio focus at all — and if so, when and which type (MAY_DUCK vs none vs a short STT-only request)?
Is there an official or semi-official API to disable SpeechRecognizer beeps across OEMs / Google engine?
For accessibility voice commands, is on-device speech recognition / AudioRecord + custom VAD a better fit than SpeechRecognizer?
What is the recommended handoff from TTS UtteranceProgressListener.onDone to STT startListening (delay, audio session, mic release)?

Constraints

Accessibility is used only for core functionality (Play policy)
We can’t ask users to “turn off YouTube” or “accept beeps”
Needs to work across Samsung, Pixel, and similar devices

I can share repro logs or a minimal sample project. Looking for real-world patterns from anyone who has shipped something similar.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the interaction on Android 14/15, including Samsung and Pixel devices, using AccessibilityService, TextToSpeech, SpeechRecognizer, and media playback. Trace UtteranceProgressListener.onDone through startListening, and compare audio focus, AudioAttributes, beep behavior, and ERROR_NO_MATCH during YouTube playback. Done means documenting or demonstrating a cross-device handoff that preserves media, suppresses or safely handles beeps, and opens the listening window reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
kotlin
Domain
accessibility, audio-video-rtc, mobile-dev
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.