[FEAT] Screen-aware transcription context
- Dominant language
- Swift
- Stars
- 2.9k
- Forks
- 226
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
Domain-specific words onl screen (code identifiers, app names, technical terms) are frequently misrecognised because the transcription engine has no context about what the user is looking at. Same class of problem as #154, but approached differently.
**Describe the solution you'd like**
Capture the active window before/during recording, run OCR via e.g. Apple Vision (VNRecognizeTextRequest), and pass extracted terms to WhisperKit's DecodingOptions. initial prompt to bias recognition. This is how Wispr Flow achieves high accuracy on domain-specific vocab.
**Describe alternatives you've considered**
- Custom dictionary / word remapping (#154): Works for known words but requires manual upkeep and doesn't adapt to current screen content. Worse UX
- Accessibility API text extraction: Read focused app content via AXUIElement; lighter than OCR but not all apps expose their content. Less impact
**Additional context**
- Whisperkit already accepts initialPrompt in DecodingOptions; partial plumbing exists.
- Parakeet (FluidAudio) has no prompt API, so this would only benefit Whisperkit unless FluidAudio adds support.
- Requires Screen Recording permission (sandbox entitlement & user consent). Could be a toggleable setting.
Would love to draft a PR if this feature request aligns with your vision for the project!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the existing WhisperKit DecodingOptions initialPrompt plumbing, then review VNRecognizeTextRequest and AXUIElement as the proposed context sources. Done would capture the active window, extract screen terms, and pass them to WhisperKit while handling Screen Recording permission and an optional setting; Parakeet is out of scope unless it gains prompt support.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, swift
- Domain
- audio-video-rtc, desktop
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100