Jordan-Hall / Jordan-Hall/browser

[P0][PC-02] Windows native-control adapter

Open
#69 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Programme: #1
Epic: #23

## Objective
Implement the first production desktop-control backend on Windows using UI Automation and platform APIs, with bounded visual fallback only where necessary.

## Scope
- Window/process enumeration and identity mapping.
- UI Automation tree extraction and stable element references where possible.
- Invoke/select/set-value/text/focus/scroll patterns.
- Application/file launch and handoff through scoped runtime capabilities.
- Bounded screenshot/visual grounding fallback for unsupported controls.
- User takeover detection and physical-input lease integration.
- Handling elevation boundaries, UAC/secure desktop, permission prompts and inaccessible controls.
- Reference workflow corpus across common native/Win32/UWP/Chromium/Electron apps.

## Safety rules
- Never automate secure-desktop approval, UAC elevation, login protection, MFA or CAPTCHA bypass.
- Re-observe state after focus/window changes.
- Visual fallback does not inherit semantic confidence.

## Acceptance criteria
- [ ] Reference Windows workflows complete using semantic UIA where supported and verify postconditions.
- [ ] UAC/secure-desktop and protected auth flows stop for user handoff.
- [ ] Changed/closed/replaced windows invalidate stale targets.
- [ ] Visual fallback is bounded and separately logged/evaluated.
- [ ] Manual user input relinquishes agent control.
- [ ] Accessibility/control failures return structured diagnostics rather than blind retries.

## Dependencies
- PC-01

**First phase:** P0
**Maturity target:** P3
**Owner:** harness-pc-providers

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing dependency PC-01 and the objective, scope, and safety rules for the Windows native-control backend. Use the acceptance criteria as the completion checklist: semantic UI Automation workflows with postcondition checks, safe user handoff for protected flows, stale-target invalidation, bounded visual fallback, input relinquishment, and structured diagnostics.

Written by the indexing model from the issue text.

Assessment

Domain
desktop, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.