hcengineering / hcengineering/platform

Feature: Enable transcription language selector in Room settings (currently commented out)

Open
#10,532 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
27.7k
Forks
2.2k
PR merge metrics
No merged PRs in 30d

Description

### Summary

The `RoomLanguageSelector` component is **fully built and functional** — supporting 45 languages with flag emojis and native labels — but the UI row is commented out in `RoomTranscriptionSettings.svelte`, making it inaccessible to users. All rooms default to `language: 'en'` at creation time with no way to change it from the UI.

This means:
- **OpenAI STT** always receives `language: "en"`, producing garbled output and low confidence scores (~30%) for non-English speakers
- **Deepgram STT** is unaffected by user choice (hardcoded to `language: "multi"` auto-detect, `updateLanguage()` is a no-op)

### Provider-specific language behavior

Even if the UI selector were enabled today, it would **only affect OpenAI STT**. Deepgram's implementation intentionally ignores the room language:

| Behavior | OpenAI STT | Deepgram STT |
|----------|-----------|--------------|
| **Language default** | `"en"` (stored on instance) | N/A — hardcoded `"multi"` in `getOptions()` |
| **`updateLanguage()` behavior** | Stores new language, sends live `transcription_session.update` to all active WebSocket connections (gated by `OPENAI_PROVIDE_LANGUAGE`) | **No-op** — method body is entirely commented out |
| **Language sent to API** | `this.language` (dynamic, from room metadata) or `undefined` if `OPENAI_PROVIDE_LANGUAGE=false` | Always `"multi"` (hardcoded), regardless of room setting |
| **Auto-detection** | Only when language is omitted (`OPENAI_PROVIDE_LANGUAGE=false`) | Always — Nova-3 `"multi"` mode handles multilingual automatically |
| **Mid-meeting language change** | Works — sends `transcription_session.update` in real-time | Would require connection restart (code exists but is commented out) |

The Deepgram `updateLanguage()` originally had a stop/restart implementation to reconnect with the new language, but it was commented out — likely because Nova-3's `"multi"` mode already handles multilingual meetings well without explicit hints.

### The infrastructure is already complete

| Layer | Status | Details |
|-------|--------|---------|
| **Types** | ✅ Done | `RoomLanguage` union type with 45 languages, `languagesDisplayData` with flag emojis |
| **Room schema** | ✅ Done | `Room.language: RoomLanguage` field persisted in DB |
| **UI component** | ✅ Done | `RoomLanguageSelector.svelte` — dropdown with search, two display modes |
| **Settings popup** | ❌ Commented out | `RoomTranscriptionSettings.svelte` lines ~34–39 |
| **Room creation default** | ❌ Hardcoded `'en'` | `AddRoomPopup.svelte` — ignores browser locale |
| **Metadata flow** | ✅ Done | `startTranscription()` → `connectMeeting(room.language)` → aibot → love → LiveKit metadata |
| **Love service** | ✅ Done | `/transcription` and `/language` endpoints update room metadata |
| **Love-agent** | ✅ Done | `RoomMetadataChanged` → `stt.updateLanguage()` |
| **OpenAI STT** | ✅ Done | `transcription_session.update` with new language, works dynamically mid-meeting |
| **Deepgram STT** | ❌ No-op | `updateLanguage()` body is commented out, `getOptions()` hardcodes `'multi'` |

### Where the code is commented out

**`plugins/love-resources/src/components/RoomTranscriptionSettings.svelte`:**

```svelte

```

**`plugins/love-resources/src/components/AddRoomPopup.svelte`:**

```ts
language: 'en', // hardcoded, no browser locale fallback
```

**`services/ai-bot/love-agent/src/deepgram/stt.ts`:**

```ts
// private language: string = 'en'

updateLanguage (language: string): void {
// const shouldRestart = (this.language ?? 'en') !== language
// this.language = language
// if (shouldRestart) {
// this.stop()
// this.start()
// }
}
```

### Impact on self-hosted deployments

Self-hosted users deploying with OpenAI STT in non-English environments get:
- Low confidence transcriptions (30–50% probability)
- Garbled text for non-English speech
- No way to fix it without modifying source code or switching to Deepgram

### Proposed fix

1. **Uncomment** the language selector row in `RoomTranscriptionSettings.svelte` and add the missing imports (`ui` plugin reference and `RoomLanguageSelector`)
2. **Default to browser locale** in `AddRoomPopup.svelte` instead of hardcoded `'en'` (fallback to `'en'` if the browser locale is not in the supported list)
3. **Restore `updateLanguage()`** in `deepgram/stt.ts` — uncomment the stop/restart implementation and wire `this.language` through `getOptions()` (default `'multi'` preserves current behaviour; setting a language overrides auto-detect for single-language meetings)

### Workarounds (until fixed)

- **Switch to Deepgram** (`STT_PROVIDER=deepgram`) — uses `language: "multi"` auto-detect, works for all languages
- **Set `OPENAI_PROVIDE_LANGUAGE=false`** on the love-agent — disables the English hint, lets OpenAI auto-detect (less accurate than an explicit language hint)
- **Direct DB update** — `UPDATE love SET data = jsonb_set(data, '{language}', '"es"') WHERE _class = 'love:class:Room'` (requires CockroachDB access)

### Environment

- **Version:** v0.7.353 (`hardcoreeng/*` images)
- **STT Provider:** OpenAI (Realtime API, `gpt-4o-transcribe`)
- **Deployment:** Self-hosted via Docker Compose

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.