googlesamples / googlesamples/mlkit
[Feature request] Add word-level timing metadata and short-utterance guidance for GenAI Speech Recognition
- Dominant language
- Kotlin
- Stars
- 4.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
**What's your feature request? Please describe.**
We are evaluating `com.google.mlkit:genai-speech-recognition:1.0.0-alpha1` for a production Android app that performs live, structured voice-command transcription from short utterances in noisy environments.
We would like ML Kit GenAI Speech Recognition to expose richer recognition metadata. The current alpha API appears to expose only `text: String` on `PartialTextResponse` and `FinalTextResponse`. For production use, we need word-level metadata such as:
- token/word text
- start/end offsets
- confidence scores, if available
- alignment/timing data for partial and final results
Without timing/alignment data, apps must synthesize word timing, which is too error-prone for workflows that need to group words into phrases and correlate recognition with VAD/audio events.
We also need documented guidance on the intended audio/session pattern for short utterances, including:
- recommended minimum clip duration
- recommended lead/tail padding
- silence handling
- endpointing behavior
- whether short session-based recognition is supported
- whether long continuous streams are the intended usage pattern
**Mobile environment**
Android
ML Kit API/version: `com.google.mlkit:genai-speech-recognition:1.0.0-alpha1`
**Additional context**
Related Issue Tracker feature request: https://issuetracker.google.com/issues/540021698
In our testing with real live-use audio replayed through `AudioSource.fromPfd()`:
- A single long speech-only stream continued receiving audio for the full stream but recognized only about 48 words from a 1,234-word expected transcript.
- Separate VAD-derived clips as individual recognizer sessions improved coverage slightly to about 119 words, but most clips returned `ERROR_TYPE_NO_SPEECH_DETECTED`.
This appears related to existing reports about `AudioSource.fromPfd()` sensitivity to silence/endpointing, but this request is broader: richer response metadata plus guidance on the intended usage pattern for short utterance recognition.
Contributor guide
No contributing guide indexed for this repository
Research direction
No repository file, test, or entry point is named. Start by reviewing the com.google.mlkit:genai-speech-recognition:1.0.0-alpha1 API and the related Issue Tracker request; completion would require supported word-level metadata and documented guidance for short-utterance audio sessions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, kotlin
- Domain
- ai, mobile
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100