googleapis / googleapis/python-genai

Vertex AI: input_audio_transcription silently ignored - zero input_transcription events emitted

Open
#2,348 2 comments 0 reactions 1 assignee Claimed by @Venkaiahbabuneelam View on GitHub
priority: p2 type: bug
Dominant language
Python
Stars
4k
Forks
1k
Avg merge
2d 11h
Merged PRs (30d)
40

Description

## Environment

- **Model:** `gemini-live-2.5-flash-native-audio`
- **Endpoint:** Vertex AI (`vertexai=True`, project-based auth via ADC)
- **SDK version:** `google-genai` v1.70.0
- **Platform:** Cloud Run (Python 3.11)

## Configuration

```python
config = types.LiveConnectConfig(
response_modalities=["AUDIO"],
input_audio_transcription=types.AudioTranscriptionConfig(),
output_audio_transcription=types.AudioTranscriptionConfig(),
# ... speech_config, tools, etc.
)

async with client.aio.live.connect(model="gemini-live-2.5-flash-native-audio", config=config) as session:
async for response in session.receive():
sc = response.server_content
# sc.output_transcription - works, text arrives reliably
# sc.input_transcription - NEVER populated, always None
```

## Problem

`input_audio_transcription=AudioTranscriptionConfig()` is accepted in `LiveConnectConfig` without any error or warning, but the Vertex AI backend **never emits `input_transcription` events** in `LiveServerContent`.

- `output_transcription` (model speech ? text): **works correctly**
- `input_transcription` (user speech ? text): **never arrives** - zero events across hundreds of sessions over multiple days
- No error, no warning, no rejection of the config - it is silently swallowed

The SDK has the field (`LiveServerContent.input_transcription: Optional[Transcription]`), the config type exists (`AudioTranscriptionConfig`), the documentation describes it - but on Vertex AI the feature simply does not function.

## Impact

We run a production full-duplex voice AI assistant on Vertex AI. We architected our turn management system around `input_transcription` events:

1. **User transcription display** - users could not see their own speech in the UI
2. **Turn semaphore** - our dead air detection relies on `input_transcription` to call `record_user_speech()`. Since it never fires, the system cannot distinguish "user is silent" from "user is speaking"
3. **Cascade failure** - broken turn detection triggered aggressive reconnection loops, wasting compute and degrading UX

We spent **8+ hours** debugging what we initially thought was Gemini session instability, deploying four production hotfixes, before tracing it to this single missing feature. The fix was a one-line fallback to browser-side `SpeechRecognition`.

## Expected behavior

If `input_audio_transcription` is set in config:
- `server_content.input_transcription` should contain user speech text (same as `output_transcription` does for model speech)

If the feature is **not supported** on Vertex AI:
- The config should be **rejected** with a clear error, not silently accepted

## Related issues

This has been reported multiple times with no clear resolution on Vertex AI:

- #735 - "why inputAudioTranscription is not available?" (response: "only supported in Vertex" - but it doesn't work there either)
- #1279 - "Audio transcription not returned in response" on Vertex AI
- googleapis/js-genai#1212 - No transcription messages in Node.js SDK
- googleapis/js-genai#1429 - Transcription `finished` flag never updates

## Ask

1. Is `input_audio_transcription` actually supported on Vertex AI for `gemini-live-2.5-flash-native-audio`?
2. If not, please reject the config or document the limitation clearly
3. If it is supposed to work, what's the timeline for a fix?

Silent acceptance of config that gets ignored is the worst possible developer experience - especially for paying enterprise customers building production infrastructure on your platform.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.