JesseRWeigel / JesseRWeigel/koe

Voice-first conversation mode (Realtime API / Grok-Voice)

Open
#39 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
2
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Context

Language learning is a voice-first domain. Text-only tutoring works but drops most of the actual skill being learned (listening, speaking, prosody, conversational flow).

ThursdAI 2026-04-23 covered xAI's **Grok-Voice-think-fast 1.0** and reminded us that OpenAI's Realtime API is GA. Both are end-to-end omni models (voice in, voice out, tool calls) rather than the traditional STT-LLM-TTS stack. Latency is sub-second. Proper nouns and code identifiers handled better.

## Proposal

Add voice-first conversation mode to Koe for Japanese, Spanish, and Brazilian Portuguese. Options:

### Option A — OpenAI Realtime API (default)
Already GA. Well-documented function calling. Lets us wire a "grammar check" tool and a "define word" tool into the voice loop.

### Option B — Grok-Voice-think-fast
Check xAI developer docs for API availability. May offer better prosody for certain languages; evaluate when available.

### Option C — Local end-to-end omni
Not realistic yet for our latency needs, but watch the space. Privacy would be a big win.

## Scope for v1

- Japanese conversation mode: user speaks in Japanese, model responds in Japanese with optional English glosses.
- Spanish and Portuguese same pattern.
- Function-calling tools: `explain_grammar(sentence)`, `define(word)`, `pronunciation_score(audio)`.
- Session transcripts logged for later review.
- A "switch difficulty" voice command.

## Risks

| Risk | Mitigation |
|------|------------|
| Cost per user hour | Per-session cap, prefer free tier for development |
| Model's accent or native speaker match quality | Manual evaluation per language before ship |
| Privacy of voice data | Store transcripts locally by default, explicit opt-in for cloud logging |

## Links

- ThursdAI 2026-04-23: https://sub.thursdai.news/p/apr-23-openais-week-gpt-55-gpt-image
- OpenAI Realtime API: https://platform.openai.com/docs/guides/realtime

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no files, tests, or entry points; start by reading the linked OpenAI Realtime API documentation and checking xAI API availability. Then map the existing conversation, session, and transcript paths before choosing a provider. Done means a scoped voice-mode implementation covering the three languages, tools, transcripts, difficulty control, cost, evaluation, and privacy.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, audio-video-rtc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.