uttrflow / uttrflow/uttrflow-swift

Nothing sets AXManualAccessibility, so apps built on a bundled browser engine publish a field only if another assistive app has already switched it on

Open
#605 0 comments 0 reactions 0 assignees View on GitHub
area:ai-suggestions enhancement help wanted P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

Desktop apps built on a bundled browser engine do not build their Accessibility tree until an assistive client asks for it. The documented way to ask is to set the `AXManualAccessibility` attribute to true on the application element. The older `AXEnhancedUserInterface` also does it, but it changes window animation and layout in some apps. Uttrflow sets neither:

```
$ git grep -n "AXManualAccessibility\|AXEnhancedUserInterface" -- Sources Docs
(no output)
```

So what these apps publish depends on whether something else on the Mac (a screen reader, a window manager, an automation tool) has already switched their tree on. The repository's own measurements disagree, and that is what you would expect if the tree is sometimes on and sometimes off:

- `Docs/input-paste-eligibility.md` records an editor of this kind that "expose[s] no focused element at all".
- `Docs/context-accessibility.md` records a chat app of this kind answering neither a window title nor a selection.
- `Docs/predict-reliability.md` records 609 turns in another app of this kind that *did* read a focused prompt box.

What depends on that tree:

- **AI suggestions.** `FocusedFieldReader.snapshot` (`Sources/UttrflowContext/FocusedFieldReader+System.swift:117-162`) returns nil with no focused field, so the turn is quiet with `nothingFocused`.
- **Dictation context.** `MacContextEngine.read` (`Sources/UttrflowContext/MacContextEngine+System.swift:40-59`) gets no title, selection or text around the caret. So there is no mid-sentence casing, and no on-screen spellings for names.
- **Insertion.** The Accessibility write has nothing to write into, and `PasteConfirmation` answers "not reported", so no paste into these apps is ever confirmed.

## Why it matters

Chat apps, note apps and code editors built this way are a large share of where people type. Whether suggestions and context work there should not depend on what else happens to be running.

## What needs measuring first

Switching the tree on costs the target app memory and CPU, sometimes a lot on large documents. It also stays on until the app quits. So this is `help wanted` for measurement, not a one-line change:

1. For two or three such apps, record the target's memory footprint and CPU during typing with the attribute unset and then set (`AXUIElementSetAttributeValue(app, "AXManualAccessibility" as CFString, kCFBooleanTrue)` from a throwaway harness, or a tiny script run by a binary that has Accessibility).
2. Run `uttrflow-dev probe surface` in each, before and after, and report whether a focused field, its value, its caret bounds and its text style appear.

## Acceptance criteria

- A documented decision, backed by the numbers above, on whether and when Uttrflow sets `AXManualAccessibility`. For example: only for the frontmost app, only while AI suggestions are on, never `AXEnhancedUserInterface`.
- If it is set: once per process, not on every read. A refusal (`kAXErrorAttributeUnsupported` from a native app) is cached so it is not asked again. The dictation and suggestion readers share the same code for this.
- `Docs/context-accessibility.md` records what is published with and without it, per app kind, and the three docs above stop disagreeing.

Contributor guide

Open the contributing guide

Research direction

Start with Docs/input-paste-eligibility.md, Docs/context-accessibility.md, Docs/predict-reliability.md, and the cited FocusedFieldReader and MacContextEngine ranges. Measure target-app CPU and memory with AXManualAccessibility unset and set, then run uttrflow-dev probe surface before and after for focused-field data, value, caret bounds, and text style. Done means the decision and per-app findings are documented in Docs/context-accessibility.md, with any setting cached and shared by the readers if adopted.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, swift
Domain
accessibility, desktop-dev
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.