anthropics / anthropics/claude-code

[FEATURE] Configurable audio transcription endpoint for dictation in Cowork on 3P

Open
#92,822 0 comments 0 reactions 0 assignees View on GitHub
area:cowork area:self-hosted-environments enhancement
Dominant language
Python
Stars
145k
Forks
23.1k
PR merge metrics
PR metrics pending

Description

### Preflight Checklist

- [x] I have searched [existing requests](https://github.com/anthropics/claude-code/issues?q=is%3Aissue%20label%3Aenhancement) and this feature hasn't been requested yet
- [x] This is a single feature request (not multiple features)

### Problem Statement

Dictation is unavailable in Claude Desktop when running in third-party inference (3P) mode. The Help Center states dictation is available in Claude Cowork, but in a 3P deployment there is no way to use it.

The underlying reason appears structural rather than a bug. In 3P mode:

User identity is local device identity only — there is no Anthropic account credential to authenticate a call to an Anthropic-hosted transcription service.
The only redirectable surface is inference. Per the gateway documentation, the required contract is POST /v1/messages with streaming and tool use, plus an optional GET /v1/models. The Anthropic Messages API has no transcription endpoint.
The configuration reference has no key for a speech-to-text endpoint. inferenceProvider, inferenceGatewayBaseUrl, inferenceCustomHeaders and the rest all describe inference only.

This is arguably correct default behavior — the stated 3P security posture is that prompts, responses, files and tool outputs go only to the configured inference endpoint. Sending microphone audio to Anthropic would break exactly that guarantee. But the result is that dictation is simply unavailable to 3P users, with no supported path to enable it against infrastructure they control.

For regulated deployments this is the group most likely to want dictation routed through their own stack, and least able to use the first-party path.

### Proposed Solution

Add managed-configuration keys for a transcription endpoint, parallel to the existing gateway inference keys — for example:



Key | Type | Description
-- | -- | --
transcriptionProvider | enum | none (default) or gateway
transcriptionBaseUrl | string | Base URL of an OpenAI-compatible transcription endpoint
transcriptionApiKey | string | Credential for that endpoint
transcriptionAuthScheme | enum | bearer (default) or x-api-key
transcriptionModel | string | Model ID the endpoint expects


Contract: POST /v1/audio/transcriptions, OpenAI-compatible multipart (file, model). Default off, so existing 3P deployments are unchanged and no audio leaves the device unless an admin opts in.


This is a small surface because the OpenAI transcription shape is already near-universal. LiteLLM — named in the gateway docs as a supported gateway — already exposes /v1/audio/transcriptions with OpenAI, Azure, Fireworks AI, Groq, Deepgram, Mistral (Voxtral) and OVHcloud backends, plus per-request fallbacks. Portkey and other gateways in this category are similar. Operators would not need to change anything server-side; the gap is purely that the client has no way to be pointed at an endpoint that already exists.


It also fits the existing security model cleanly: audio would go to the same operator-controlled infrastructure as inference, keeping the "only your configured endpoint" property intact, and would be auditable in the same gateway logs.

### Alternative Solutions

OS-level dictation (macOS Fn Fn, Windows Win+H) — works today and injects into the Cowork composer. This is the current workaround, but accuracy and language coverage are weaker than a Whisper-class model, it varies by platform, and it can't be centrally configured or audited by an admin.

Local Whisper with a hotkey (whisper.cpp, MacWhisper) — better accuracy, but per-user setup with no MDM story, and duplicates capacity the org's gateway may already provide.

Routing transcription through Anthropic in 3P mode — rejected, for the data-residency reasons above.
Reusing inferenceGatewayBaseUrl and assuming the gateway also serves /v1/audio/transcriptions — simpler, but conflates two services that are often separate deployments, and would break gateways serving only the Messages API. Separate keys seem safer.

### Priority

Critical - Blocking my work

### Feature Category

Interactive mode (TUI)

### Use Case Example

_No response_

### Additional Context

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the configuration reference and the existing inferenceProvider, inferenceGatewayBaseUrl, inferenceCustomHeaders, and related managed-configuration handling. Trace how Cowork dictation currently selects its transcription service, then define the integration boundary for the proposed keys and POST /v1/audio/transcriptions contract. Done means opt-in configuration, supported authentication and model settings, and no audio routing by default.

Written by the indexing model from the issue text.

Assessment

Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.