uttrflow / uttrflow/uttrflow-swift
Nothing sets AXManualAccessibility, so apps built on a bundled browser engine publish a field only if another assistive app has already switched it on
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## What happens
Desktop apps built on a bundled browser engine do not build their Accessibility tree until an assistive client asks for it. The documented way to ask is to set the `AXManualAccessibility` attribute to true on the application element. The older `AXEnhancedUserInterface` also does it, but it changes window animation and layout in some apps. Uttrflow sets neither:
```
$ git grep -n "AXManualAccessibility\|AXEnhancedUserInterface" -- Sources Docs
(no output)
```
So what these apps publish depends on whether something else on the Mac (a screen reader, a window manager, an automation tool) has already switched their tree on. The repository's own measurements disagree, and that is what you would expect if the tree is sometimes on and sometimes off:
- `Docs/input-paste-eligibility.md` records an editor of this kind that "expose[s] no focused element at all".
- `Docs/context-accessibility.md` records a chat app of this kind answering neither a window title nor a selection.
- `Docs/predict-reliability.md` records 609 turns in another app of this kind that *did* read a focused prompt box.
What depends on that tree:
- **AI suggestions.** `FocusedFieldReader.snapshot` (`Sources/UttrflowContext/FocusedFieldReader+System.swift:117-162`) returns nil with no focused field, so the turn is quiet with `nothingFocused`.
- **Dictation context.** `MacContextEngine.read` (`Sources/UttrflowContext/MacContextEngine+System.swift:40-59`) gets no title, selection or text around the caret. So there is no mid-sentence casing, and no on-screen spellings for names.
- **Insertion.** The Accessibility write has nothing to write into, and `PasteConfirmation` answers "not reported", so no paste into these apps is ever confirmed.
## Why it matters
Chat apps, note apps and code editors built this way are a large share of where people type. Whether suggestions and context work there should not depend on what else happens to be running.
## What needs measuring first
Switching the tree on costs the target app memory and CPU, sometimes a lot on large documents. It also stays on until the app quits. So this is `help wanted` for measurement, not a one-line change:
1. For two or three such apps, record the target's memory footprint and CPU during typing with the attribute unset and then set (`AXUIElementSetAttributeValue(app, "AXManualAccessibility" as CFString, kCFBooleanTrue)` from a throwaway harness, or a tiny script run by a binary that has Accessibility).
2. Run `uttrflow-dev probe surface` in each, before and after, and report whether a focused field, its value, its caret bounds and its text style appear.
## Acceptance criteria
- A documented decision, backed by the numbers above, on whether and when Uttrflow sets `AXManualAccessibility`. For example: only for the frontmost app, only while AI suggestions are on, never `AXEnhancedUserInterface`.
- If it is set: once per process, not on every read. A refusal (`kAXErrorAttributeUnsupported` from a native app) is cached so it is not asked again. The dictation and suggestion readers share the same code for this.
- `Docs/context-accessibility.md` records what is published with and without it, per app kind, and the three docs above stop disagreeing.
Contributor guide
Research direction
Start with Docs/input-paste-eligibility.md, Docs/context-accessibility.md, Docs/predict-reliability.md, and the cited FocusedFieldReader and MacContextEngine ranges. Measure target-app CPU and memory with AXManualAccessibility unset and set, then run uttrflow-dev probe surface before and after for focused-field data, value, caret bounds, and text style. Done means the decision and per-app findings are documented in Docs/context-accessibility.md, with any setting cached and shared by the readers if adopted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, swift
- Domain
- accessibility, desktop-dev
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100