GoogleCloudPlatform / GoogleCloudPlatform/generative-ai
[Bug]: [Gemini TTS] Streaming Returns LINEAR16 PCM When OGG_OPUS is Requested
- Dominant language
- Jupyter Notebook
- Stars
- 17.7k
- Forks
- 4.5k
- Avg merge
- 12h 38m
- Merged PRs (30d)
- 42
Description
### File Name
N/A
### What happened?
When using Google Cloud Text-to-Speech API's streaming synthesis with `OGG_OPUS` audio encoding, **Chirp3-HD** and **Gemini TTS** models behave completely differently:
- ✅ **Chirp3-HD**: Returns proper OGG container format with OPUS codec (as requested)
- ❌ **Gemini TTS**: Returns **LINEAR16 PCM** instead, ignoring the encoding parameter
This is a critical bug where Gemini TTS **silently ignores the requested audio encoding** and returns uncompressed PCM data instead of the requested compressed OGG_OPUS format.
## Test Results
### Chirp3-HD (Working as Expected)
```
Packets: 4
Total size: 11,046 bytes
First packet: Starts with 'OggS' ✅
Contains 'OpusHead': Yes ✅
```
**Hex dump (first 20 bytes):**
```
4f67675300020000000000000000000000000000
^OggS (proper OGG magic number)
```
### Gemini TTS (Issue)
```
Packets: 121
Total size: 231,914 bytes
First packet: Starts with 'OggS' ❌
Contains 'OpusHead': No ❌
Packet size: 1920 bytes (consistent)
```
**Hex dump (first 20 bytes):**
```
08000b00fcffe9ffdfffdbffd2ffc6ffc7ffdfff
(This is LINEAR16 PCM audio data!)
```
**Analysis:**
- 1920 bytes per packet = 960 samples × 2 bytes (16-bit)
- 960 samples at 24kHz = 40ms of audio
- Data plays correctly as PCM: `ffplay -f s16le -ar 24000 gemini_output.raw` ✅
- This is **LINEAR16 PCM**, NOT OGG_OPUS as requested!
## Expected Behavior
When requesting `AudioEncoding.OGG_OPUS`, the API should return audio data wrapped in an OGG container format with OPUS codec, regardless of which TTS model is used.
## Actual Behavior
Gemini TTS models **completely ignore the audio encoding parameter** and return LINEAR16 PCM data instead. The API silently substitutes a different format without any warning or error.
## Configuration Used
```python
voice = texttospeech.VoiceSelectionParams(
language_code="en-US",
name="Kore",
model_name="gemini-2.5-flash-preview-tts",
)
streaming_config = texttospeech.StreamingSynthesizeConfig(
voice=voice,
streaming_audio_config=texttospeech.StreamingAudioConfig(
audio_encoding=texttospeech.AudioEncoding.OGG_OPUS,
sample_rate_hertz=24000,
),
)
```
## Impact
1. **Silent format mismatch**: Application expects OGG_OPUS but receives LINEAR16 PCM
2. **Bandwidth waste**: LINEAR16 PCM is ~5-10x larger than OPUS compressed audio
3. **Breaking compatibility**: Cannot integrate with systems expecting OGG_OPUS format
4. **Model-specific code**: Requires completely different handling for Gemini vs Chirp3
5. **No error or warning**: API silently returns wrong format without indication
6. **Cost implications**: Larger data transfers increase bandwidth costs and latency
## Packet Size Analysis
| Model | Packet Count | Avg Packet Size | Format | Actual Format |
| ---------- | ------------ | --------------- | ----------------------------- | ------------- |
| Chirp3-HD | 4 | 2,761 bytes | Variable (proper OGG pages) | OGG_OPUS ✅ |
| Gemini TTS | 121 | 1,917 bytes | Fixed 1920 bytes (PCM frames) | LINEAR16 ❌ |
The fixed 1920-byte packets from Gemini TTS equal exactly 40ms of LINEAR16 PCM audio (960 samples × 2 bytes at 24kHz). Chirp3-HD's variable packet sizes indicate proper OGG page segmentation with OPUS compression.
## Reproduction Steps
1. Use the Google Cloud Text-to-Speech API with streaming synthesis
2. Configure `AudioEncoding.OGG_OPUS` with 24kHz sample rate
3. Compare output from `en-US-Chirp3-HD-Charon` vs `gemini-2.5-flash-preview-tts`
4. Inspect the first packet's magic bytes
**Test script:** `test_chirp3_vs_gemini.py` (included in this repository)
To reproduce:
```bash
python test_chirp3_vs_gemini.py
```
[test_chirp3_vs_gemini.py](https://github.com/user-attachments/files/23442894/test_chirp3_vs_gemini.py)
This will generate two files:
- `test_chirp3_streaming.ogg` (proper OGG_OPUS, playable with `ffplay`) ✅
- `test_gemini_streaming.ogg` (actually LINEAR16 PCM, play with `ffplay -f s16le -ar 24000`) ❌
## Environment
- **Library**: `google-cloud-texttospeech`
- **API**: Google Cloud Text-to-Speech Streaming API
- **Models tested**:
- `en-US-Chirp3-HD-Charon` ✅
- `gemini-2.5-flash-preview-tts`
❌
- **Python**: 3.13.7
- **Date**: November 10, 2025
## Additional Findings
- **Gemini TTS does NOT support LINEAR16 encoding**: Requesting `AudioEncoding.LINEAR16` returns error: `400 Unsupported audio encoding`
- **Yet it returns LINEAR16 anyway**: Despite not officially supporting it, this is what you actually get when requesting OGG_OPUS
- **No documentation**: The API doesn't document what encodings Gemini TTS actually supports
- **Silent failure**: No error, warning, or indication that the requested format is unavailable
## Request
Please fix Gemini TTS to either:
1. **Preferred**: Support `OGG_OPUS` encoding properly (return OGG containers with OPUS codec), consistent with Chirp3-HD
2. **Alternative**: Return a clear error when OGG_OPUS is requested if not supported, rather than silently returning a different format
3. **Documentation**: Clearly document which audio encodings are supported by each model
## Additional Notes
- The issue affects real-time streaming scenarios where bandwidth and format compliance are critical
- Standards compliance: [RFC 7845 - Ogg Encapsulation for the Opus Audio Codec](https://datatracker.ietf.org/doc/html/rfc7845)
- Streaming protocols (WebRTC, etc.) expect proper OGG_OPUS format
- LINEAR16 PCM uses ~10x more bandwidth than OPUS for the same audio quality
### Relevant log output
```shell
```
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Contributor guide
Research direction
Start by running the attached test_chirp3_vs_gemini.py with the documented Gemini TTS and Chirp3-HD configurations, then inspect the generated stream headers and packet sizes. Done means Gemini TTS returns OGG_OPUS as requested, or the API returns a clear unsupported-encoding error and documents the model's supported encodings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- api, audio-video-rtc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100