feat(Chat): Text-to-Speech — speak LLM responses back to the user
- Dominant language
- TypeScript
- Stars
- 13.1k
- Forks
- 1.1k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 669
Description
## Summary
Add text-to-speech support for chat — speak LLM responses back to the user via the browser SpeechSynthesis API. Enables a full voice conversation loop: dictate → LLM responds → response spoken aloud → dictate again.
## Motivation
With dictation (#1308) handling voice input, the missing piece is voice output. Together they create a hands-free conversation mode — useful for accessibility, multitasking, or when typing isn't practical.
The Web SpeechSynthesis API is built into all modern browsers (better support than SpeechRecognition — works in Firefox too).
## Design
### `useXDSChatSpeech` hook
Headless hook wrapping the SpeechSynthesis API:
```ts
interface UseXDSChatSpeechOptions {
/** Preferred voice — matched by name or lang. @default system default */
voice?: string;
/** Speech rate, 0.5-2. @default 1 */
rate?: number;
/** Pitch, 0-2. @default 1 */
pitch?: number;
/** Volume, 0-1. @default 1 */
volume?: number;
/** Called when speech starts */
onStart?: () => void;
/** Called when speech ends */
onEnd?: () => void;
/** Called on error */
onError?: (error: SpeechSynthesisErrorEvent) => void;
}
interface UseXDSChatSpeechReturn {
/** Whether the browser supports SpeechSynthesis */
isSupported: boolean;
/** Whether currently speaking */
isSpeaking: boolean;
/** Available voices */
voices: SpeechSynthesisVoice[];
/** Speak text — queues if already speaking */
speak: (text: string) => void;
/** Stop speaking immediately */
stop: () => void;
/** Pause speech */
pause: () => void;
/** Resume paused speech */
resume: () => void;
}
```
### `XDSChatSpeechButton`
A button for assistant messages — reads the message aloud:
```tsx
{content}
}
/>
```
Button states:
- Idle: speaker icon
- Speaking: animated speaker icon (sound waves) — click to stop
- Not supported: hidden
### Conversation mode
When both dictation and speech are active, enable a continuous loop:
```tsx
const dictation = useXDSChatDictation({ ... });
const speech = useXDSChatSpeech({
onEnd: () => {
// Response finished speaking — auto-start listening again
if (conversationMode) dictation.start();
},
});
// When LLM response streams in, auto-speak it
useEffect(() => {
if (conversationMode && latestResponse) {
speech.speak(latestResponse);
}
}, [latestResponse]);
```
This is composable — consumers wire it up, XDS doesn't force the pattern.
## SpeechSynthesis API — Key Features
### Core
- `SpeechSynthesisUtterance` — the text to speak, with voice/rate/pitch/volume
- `speechSynthesis.speak(utterance)` — queue speech
- `speechSynthesis.cancel()` — stop immediately
- `speechSynthesis.pause()` / `speechSynthesis.resume()`
- `speechSynthesis.getVoices()` — list available voices (varies by OS/browser)
### Events
- `start` / `end` — utterance lifecycle
- `pause` / `resume`
- `boundary` — word/sentence boundaries (useful for highlighting text as it's read)
- `error` — synthesis failed
### Streaming consideration
LLM responses stream in. Options:
1. **Wait for full response** then speak — simplest, but slow UX
2. **Speak sentence by sentence** — split on `.!?` as text streams, queue each sentence
3. **Speak paragraph by paragraph** — split on double newlines
Option 2 is the best UX — speech starts quickly while the response is still generating.
## Browser Support
| Browser | SpeechSynthesis | Notes |
|---|---|---|
| Chrome / Edge | Full | Many voices, good quality |
| Safari | Full | Fewer voices, works well |
| Firefox | Full | Supported since Firefox 49 |
Much broader than SpeechRecognition — no flags needed anywhere.
## Implementation Plan
### Phase 1: Hook + Button
- [ ] `useXDSChatSpeech` hook
- [ ] `XDSChatSpeechButton` component
- [ ] Add `speaker` icon to registry
- [ ] Voice selector component (optional — dropdown of available voices)
- [ ] Storybook stories
- [ ] Unit tests
### Phase 2: Streaming speech
- [ ] Sentence-level chunking for streaming responses
- [ ] Queue management — cancel current speech when new message arrives
- [ ] Word boundary events for text highlighting
### Phase 3: Conversation mode
- [ ] Auto-speak assistant responses when dictation is active
- [ ] Auto-resume dictation when speech finishes
- [ ] Visual indicator for conversation mode (e.g. badge on composer)
- [ ] Push-to-talk alternative (hold button to speak, release to send)
## Open Questions
1. **Sentence chunking** — split on `.!?` or use a smarter heuristic? Code blocks and URLs contain periods.
2. **Voice selection UX** — dropdown in the composer header? Per-message? Global setting?
3. **Interruption** — if user starts speaking while AI is talking, should it stop? (Yes, probably.)
4. **Markdown stripping** — speak plain text, not `**bold**` or `[links](url)`. Need a text extraction pass.
## References
- [MDN: SpeechSynthesis](https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesis)
- [MDN: SpeechSynthesisUtterance](https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesisUtterance)
- Depends on #1308 (dictation) for the full conversation loop
## Future: Background Voice Mode
_Experimental — a composition pattern, not a core component._
When the user starts dictation in a chat tab then switches to another tab, the chat becomes a **background voice channel**: dictation stays active, responses are auto-spoken, and the user can work in another tab while having a voice conversation with the AI.
**Signals:**
- `document.visibilityState === 'hidden'` → tab is backgrounded
- `visibilitychange` event → tab focus changed
**Behavior:**
| Tab state | Dictation | Speech | UX |
|---|---|---|---|
| **Visible** | Manual (button) | Off (user reads) | Normal chat |
| **Hidden** | Stays active | Auto-speak responses | Voice assistant mode |
| **Returns visible** | Unchanged | Stop speaking | Back to reading |
**Sketch:**
```tsx
const [autoSpeak, setAutoSpeak] = useState(false);
useEffect(() => {
const handler = () => {
if (document.hidden) {
setAutoSpeak(true);
} else {
speech.stop();
setAutoSpeak(false);
}
};
document.addEventListener('visibilitychange', handler);
return () => document.removeEventListener('visibilitychange', handler);
}, []);
// Auto-speak new responses when backgrounded
useEffect(() => {
if (autoSpeak && latestResponse) {
speech.speak(latestResponse);
}
}, [autoSpeak, latestResponse]);
```
**Considerations:**
- Audio cue when response is ready (reuse the plop sounds from dictation)
- "Send" via voice command — detect "send" / "submit" in the transcript and auto-submit
- Browser may throttle timers in background tabs — SpeechRecognition and SpeechSynthesis should still work since they're OS-level APIs, not JS timers
- Privacy: microphone stays active in background — need clear visual indicator (browser shows mic icon in tab)
- This is a recipe/example, not a library feature — the building blocks (dictation + speech + visibilitychange) compose naturally
**Send trigger — when to submit the dictated input:**
The key insight: switch to `continuous: false` in background mode. SpeechRecognition's built-in voice activity detection handles the rest:
1. User speaks → recognition captures the utterance
2. User pauses → browser detects end of speech → fires final result + `end` event
3. `onEnd` → auto-submit the accumulated transcript → clear input
4. AI processes → response streams → auto-spoken back
5. Speech finishes → `onEnd` → restart recognition for next turn
Each utterance = one message turn. No wake words, no silence timers.
```tsx
// Background voice mode
const dictation = useXDSChatDictation({
continuous: !isBackgrounded, // continuous when typing, single-shot when voice-only
onEnd: () => {
if (isBackgrounded && inputHasContent) {
handleSubmit(); // auto-send on speech end
// Recognition restarts after AI response finishes speaking
}
},
});
```
For visible mode, `continuous: true` stays on — the user controls send with Enter/button as normal.
Contributor guide
Assessment
This issue has not been assessed yet.