JesseRWeigel / JesseRWeigel/koe
Voice-first conversation mode (Realtime API / Grok-Voice)
- Dominant language
- TypeScript
- Stars
- 2
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## Context
Language learning is a voice-first domain. Text-only tutoring works but drops most of the actual skill being learned (listening, speaking, prosody, conversational flow).
ThursdAI 2026-04-23 covered xAI's **Grok-Voice-think-fast 1.0** and reminded us that OpenAI's Realtime API is GA. Both are end-to-end omni models (voice in, voice out, tool calls) rather than the traditional STT-LLM-TTS stack. Latency is sub-second. Proper nouns and code identifiers handled better.
## Proposal
Add voice-first conversation mode to Koe for Japanese, Spanish, and Brazilian Portuguese. Options:
### Option A — OpenAI Realtime API (default)
Already GA. Well-documented function calling. Lets us wire a "grammar check" tool and a "define word" tool into the voice loop.
### Option B — Grok-Voice-think-fast
Check xAI developer docs for API availability. May offer better prosody for certain languages; evaluate when available.
### Option C — Local end-to-end omni
Not realistic yet for our latency needs, but watch the space. Privacy would be a big win.
## Scope for v1
- Japanese conversation mode: user speaks in Japanese, model responds in Japanese with optional English glosses.
- Spanish and Portuguese same pattern.
- Function-calling tools: `explain_grammar(sentence)`, `define(word)`, `pronunciation_score(audio)`.
- Session transcripts logged for later review.
- A "switch difficulty" voice command.
## Risks
| Risk | Mitigation |
|------|------------|
| Cost per user hour | Per-session cap, prefer free tier for development |
| Model's accent or native speaker match quality | Manual evaluation per language before ship |
| Privacy of voice data | Store transcripts locally by default, explicit opt-in for cloud logging |
## Links
- ThursdAI 2026-04-23: https://sub.thursdai.news/p/apr-23-openais-week-gpt-55-gpt-image
- OpenAI Realtime API: https://platform.openai.com/docs/guides/realtime
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no files, tests, or entry points; start by reading the linked OpenAI Realtime API documentation and checking xAI API availability. Then map the existing conversation, session, and transcript paths before choosing a provider. Done means a scoped voice-mode implementation covering the three languages, tools, transcripts, difficulty control, cost, evaluation, and privacy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, audio-video-rtc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100