uttrflow / uttrflow/uttrflow-swift

Typed completions post key code 0 with the text attached, so an app that reads the key code gets a run of the letter a

Open
#606 0 comments 0 reactions 0 assignees View on GitHub
area:ai-suggestions bug help wanted P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

When the Accessibility write cannot place an accepted suggestion, `CGEventTypist.type` (`Sources/UttrflowInput/SystemInput.swift:114-126`) types it:

```swift
try postTaggedKeyPair(from: source, keyCode: 0) { event in
event.flags = []
event.keyboardSetUnicodeString(stringLength: chunk.count, unicodeString: chunk)
}
```

Key code 0 is the physical A key. Each event carries up to 16 UTF-16 units of text alongside it. An app that takes the attached string gets the right characters. An app that works from the key code gets an `a` (or whatever that position types on the user's layout) for each event, or ignores the string after its first character.

The apps most likely to do that are the ones this route exists for, because they are the apps where the Accessibility write fails:

- terminal emulators that encode key presses themselves;
- IDEs built on a cross-platform UI toolkit with their own key handling;
- remote desktop and virtual machine windows, which forward key codes to another machine and have no way to send it a string;
- games and full-screen apps with raw key input.

Nothing chooses the route by what the app is. `TypedTextInsertionEngine.canInsert` (`Sources/UttrflowInput/TypedTextInsertionEngine.swift:25`) is true for anything that is not Uttrflow, and `CompletionRoute` reports success as soon as the events are posted.

## Why it matters

A wrong completion is bad. One that types `aaaa` into a terminal or a remote shell is worse: in a terminal the line may be run, and Tab or → was the only thing the user pressed. On the typed route the undo cost is also one ⌘Z per character (`Docs/predict-accept.md`).

## How to reproduce

Needs real apps, so `help wanted`. For each terminal, IDE, remote desktop or VM window you have:

1. Turn AI suggestions on, type into the app until a ghost appears, and accept it.
2. Report exactly what arrived. Also report whether `uttrflow-dev probe surface` in that field reports a value and a caret, which decides whether the Accessibility route was tried first.

## Acceptance criteria

- Each typed event carries a key code that agrees with its text where one exists (for example, the key that types that character on the current layout, looked up the way #600 proposes), or key code 0 is shown to be harmless in every app the matrix covers.
- Apps that cannot take string-carrying events (from the compatibility matrix) get no typed route: the suggestion is not offered there, and the ghost is not drawn for a completion that cannot be placed.
- `Docs/input-synthetic-keystrokes.md` says which apps read the string and which read the key code.

Contributor guide

Open the contributing guide

Research direction

Start with CGEventTypist.type in Sources/UttrflowInput/SystemInput.swift:114-126 and TypedTextInsertionEngine.canInsert in Sources/UttrflowInput/TypedTextInsertionEngine.swift:25, then review the key-code lookup proposed in #600. Reproduce the behavior with terminal, IDE, remote desktop, and VM apps, using uttrflow-dev probe surface where applicable. Done means the acceptance criteria are met and Docs/input-synthetic-keystrokes.md documents the compatibility findings.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, swift
Domain
accessibility, desktop
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.