cloudflare / cloudflare/meet

Feature Suggestion: Add AI subtitles

Open
#147 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
2.3k
Forks
348
PR merge metrics
No merged PRs in 30d

Description

Having subtitles would be great for language learning...

According to [the documentation](https://platform.openai.com/docs/guides/realtime-model-capabilities), you can receive real-time transcripts of the audio through the `response.audio_transcript.delta` server events. This happens concurrently while receiving the audio stream.

For WebRTC connections, the documentation mentions that during a session you'll receive:
- `input_audio_buffer.speech_started` events when input starts
- `input_audio_buffer.speech_stopped` events when input stops
- `response.audio_transcript.delta` events for the in-progress audio transcript
- `response.done` event when the model has completed transcribing and sending a response

This means you can get word-by-word transcription updates as the audio is being processed, allowing you to build features like real-time captions or text displays alongside the voice interaction.

The transcription events are part of the standard event lifecycle whether you're using WebRTC or WebSocket connections, so you'll have access to the transcript regardless of which connection method you choose.

We can probably create a component like this:

```
import React, { useState, useEffect } from 'react';
import { useRoomContext } from '~/hooks/useRoomContext';
import type { ClientMessage, User } from '~/types/Messages';

const AiSubtitles = () => {
const [subtitles, setSubtitles] = useState('');
const [isVisible, setIsVisible] = useState(false);
const { room } = useRoomContext();

// Record AI speech activity
const recordActivity = (user: User) => {
if (user.id === 'ai' && user.speaking) {
setIsVisible(true);
// Here we'd need the actual transcript from the AI service
// For now, we'll just show a speaking indicator
setSubtitles("AI is speaking...");
} else {
setIsVisible(false);
setSubtitles('');
}
};

if (!isVisible) return null;

return (



{subtitles}


);
};

export default AiSubtitles;
```

to support subtitles, and render it by processing the realtime API response and including this component in `/app/routes/_room.$roomName.room.tsx`

Contributor guide

Open the contributing guide

Research direction

Start in /app/routes/_room.$roomName.room.tsx and trace how realtime API responses and room activity are handled. Review the referenced OpenAI transcription events and determine how their deltas map to the proposed AiSubtitles component. Done means AI speech produces readable, updating subtitles in the room and the display ends when the response completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
react, typescript
Domain
audio-video-rtc, frontend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.