googlesamples / googlesamples/mlkit

[Feature request] Add word-level timing metadata and short-utterance guidance for GenAI Speech Recognition

Open
#1,066 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Kotlin
Stars
4.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

**What's your feature request? Please describe.**

We are evaluating `com.google.mlkit:genai-speech-recognition:1.0.0-alpha1` for a production Android app that performs live, structured voice-command transcription from short utterances in noisy environments.

We would like ML Kit GenAI Speech Recognition to expose richer recognition metadata. The current alpha API appears to expose only `text: String` on `PartialTextResponse` and `FinalTextResponse`. For production use, we need word-level metadata such as:

- token/word text
- start/end offsets
- confidence scores, if available
- alignment/timing data for partial and final results

Without timing/alignment data, apps must synthesize word timing, which is too error-prone for workflows that need to group words into phrases and correlate recognition with VAD/audio events.

We also need documented guidance on the intended audio/session pattern for short utterances, including:

- recommended minimum clip duration
- recommended lead/tail padding
- silence handling
- endpointing behavior
- whether short session-based recognition is supported
- whether long continuous streams are the intended usage pattern

**Mobile environment**

Android

ML Kit API/version: `com.google.mlkit:genai-speech-recognition:1.0.0-alpha1`

**Additional context**

Related Issue Tracker feature request: https://issuetracker.google.com/issues/540021698

In our testing with real live-use audio replayed through `AudioSource.fromPfd()`:

- A single long speech-only stream continued receiving audio for the full stream but recognized only about 48 words from a 1,234-word expected transcript.
- Separate VAD-derived clips as individual recognizer sessions improved coverage slightly to about 119 words, but most clips returned `ERROR_TYPE_NO_SPEECH_DETECTED`.

This appears related to existing reports about `AudioSource.fromPfd()` sensitivity to silence/endpointing, but this request is broader: richer response metadata plus guidance on the intended usage pattern for short utterance recognition.

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository file, test, or entry point is named. Start by reviewing the com.google.mlkit:genai-speech-recognition:1.0.0-alpha1 API and the related Issue Tracker request; completion would require supported word-level metadata and documented guidance for short-utterance audio sessions.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, kotlin
Domain
ai, mobile
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.