kitlangton / kitlangton/Hex

[FEAT] Screen-aware transcription context

Open
#187 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
Swift
Stars
2.9k
Forks
226
PR merge metrics
No merged PRs in 30d

Description

**Is your feature request related to a problem? Please describe.**
Domain-specific words onl screen (code identifiers, app names, technical terms) are frequently misrecognised because the transcription engine has no context about what the user is looking at. Same class of problem as #154, but approached differently.

**Describe the solution you'd like**
Capture the active window before/during recording, run OCR via e.g. Apple Vision (VNRecognizeTextRequest), and pass extracted terms to WhisperKit's DecodingOptions. initial prompt to bias recognition. This is how Wispr Flow achieves high accuracy on domain-specific vocab.

**Describe alternatives you've considered**
- Custom dictionary / word remapping (#154): Works for known words but requires manual upkeep and doesn't adapt to current screen content. Worse UX
- Accessibility API text extraction: Read focused app content via AXUIElement; lighter than OCR but not all apps expose their content. Less impact

**Additional context**
- Whisperkit already accepts initialPrompt in DecodingOptions; partial plumbing exists.
- Parakeet (FluidAudio) has no prompt API, so this would only benefit Whisperkit unless FluidAudio adds support.
- Requires Screen Recording permission (sandbox entitlement & user consent). Could be a toggleable setting.

Would love to draft a PR if this feature request aligns with your vision for the project!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing WhisperKit DecodingOptions initialPrompt plumbing, then review VNRecognizeTextRequest and AXUIElement as the proposed context sources. Done would capture the active window, extract screen terms, and pass them to WhisperKit while handling Screen Recording permission and an optional setting; Parakeet is out of scope unless it gains prompt support.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, swift
Domain
audio-video-rtc, desktop
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.