Whisper realtime translates non-English speech despite 99-language claim

Impact

This is currently blocking the multilingual launch of our production macOS dictation application. The realtime openai/whisper-large-v3 endpoint translates non-English speech into English instead of transcribing it in the original language.

Primary reproduction: raw WebSocket protocol

Using Together's documented realtime WebSocket protocol:

1. Connect to:
   wss://api.together.ai/v1/realtime?intent=transcription&model=openai/whisper-large-v3&input_audio_format=pcm_s16le_16000&turn_detection=none
2. After session.created, send:

   {"type":"transcription_session.updated","session":{"language":"auto"}}
   

3. Append 16 kHz mono PCM16 audio and commit the buffer.
4. Repeat with language: "es".

For audio saying:

Hola, buenos días.

Both Auto and explicit Spanish return:

Hello, good morning.

Independent confirmation

I reproduced the same behavior with Together's official Python SDK v2.28.0 using client.beta.realtime.transcription(...), with both language="auto" and language="es". A second Spanish recording returned an empty result when Spanish was explicitly selected.

Controls

The same recordings are transcribed correctly by:

• Together Parakeet realtime streaming
• Together Whisper REST with explicit language="es"

This isolates the problem to the Together Whisper realtime path rather than the audio or client encoding.

Expected behavior

Realtime Whisper should preserve the source language, consistent with the model page's claim that streaming Whisper supports 99 languages:

https://www.together.ai/models/whisper-large-v3-streaming

Requested action

Please confirm whether realtime Whisper is currently English-only, fix the endpoint or correct the documentation, and provide an ETA or production-safe workaround. I can provide the original recordings and complete WebSocket event logs immediately.

@yadavsahil197 @sbeurnier @zainhas @blainekasten — tagging the public author and reviewers of the realtime SDK implementation because this is critical for our launch.

Start by reproducing the raw WebSocket flow and the Python SDK call with the supplied Spanish recordings or event logs. Compare both realtime paths with Together Whisper REST and Parakeet, then trace the realtime transcription implementation; done means the endpoint preserves Spanish, or the documentation clearly states the limitation and gives a production-safe workaround.