google-gemini / google-gemini/gemini-cli

bug: voice providers resolve connect before transcription is ready

Open
#28,647 1 comment 0 reactions 0 assignees View on GitHub
area/extensions effort/medium priority/p2 status/bot-triaged
Dominant language
TypeScript
Stars
107k
Forks
14.6k
Avg merge
2d 3h
Merged PRs (30d)
45

Description

## What happened?

The voice transcription providers can resolve `connect()` before they are usable, causing callers to start recording and send audio to a dead or incompletely initialized provider.

Two implementations exhibit the same contract problem:

### Local Whisper

In `packages/core/src/voice/whisperTranscriptionProvider.ts:70-151`, readiness normally waits for the `main: processing` stderr marker. However:

- The child `close` handler does not reject a still-pending connection.
- A ten-second fallback resolves unconditionally if readiness was never observed.
- The timeout is not cleared after success, error, or close.

If `whisper-stream` exits before becoming ready without emitting an `error` event or an stderr line containing lowercase `error`, `connect()` still resolves ten seconds later even though `this.process` is already null.

### Gemini Live

In `packages/core/src/voice/geminiLiveTranscriptionProvider.ts:66-144`, `connect()` waits only for the WebSocket `open` event, sends the setup message, and returns immediately. It does not wait for the protocol's setup-complete response. The message handler processes only `serverContent`, so setup acknowledgement/rejection is not part of connection readiness.

## How can this be reproduced?

- Whisper: mock the spawned child to emit `close` before the readiness marker, advance timers ten seconds, and observe `connect()` resolve.
- Gemini Live: mock a socket that emits `open` but never sends setup completion, and observe `connect()` resolve immediately after `send()`.

The voice hook starts the recorder immediately after awaiting `connect()`, making this externally visible as lost initial audio or a connected UI backed by no live provider.

## What did you expect to happen?

`connect()` should resolve only when the provider is ready to accept audio. It should reject on close/error/setup rejection before readiness, and all readiness timers/listeners should be cleaned up on every terminal path.

## Why this matters

The current behavior produces silent voice failures and misleading connected state, which is especially confusing because the UI has already completed model/provider setup.

## Suggested direction

- Define readiness semantics explicitly in the `TranscriptionProvider` contract.
- Reject Whisper connection on pre-ready close and make timeout a rejection, not success.
- Wait for Gemini Live setup completion before resolving.
- Add provider lifecycle tests covering ready, early close, error, and timeout.

## Client information

Found by source audit on current `main`, commit `f47d6c6f7`. Whisper currently has only an unavailable-binary test; Gemini Live has no dedicated provider test.

Contributor guide

Open the contributing guide

Research direction

Read packages/core/src/voice/whisperTranscriptionProvider.ts:70-151 and packages/core/src/voice/geminiLiveTranscriptionProvider.ts:66-144, then inspect the TranscriptionProvider contract and existing Whisper unavailable-binary test. Add lifecycle coverage for ready, early close, error, setup rejection, and timeout paths; done means connect() resolves only after readiness and rejects or cleans up on every failed terminal path.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
audio-video-rtc, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.