apache / apache/maka

feat(linux): add Linux computer-use backend (AT-SPI/X11 accessibility)

Open
#3,891 0 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
TypeScript
Stars
5.4k
Forks
502
Avg merge
1d 2h
Merged PRs (30d)
715

Description

## Summary

The Computer Use subsystem (packages/computer-use) is entirely macOS-native. selectComputerUseBackend() returns NONE for any non-Darwin platform, and the maka-cu executor references macOS-specific APIs (AXRaise, AXMenu, ScreenCaptureKit, macOS seatbelt).

## Current State

- **packages/computer-use/src/select-backend.ts:106**: Hardcoded if (process.platform !== 'darwin') return NONE
- **packages/computer-use/src/maka-cu-backend.ts**: 1300+ lines of macOS-specific code
- **pps/desktop/src/main/computer-use/**: 7 files with macOS-only Electron desktop features (PIP windows, cursor overlay, screen lock)

## Proposed Solution

1. Create a platform abstraction layer in packages/computer-use to decouple macOS-specific logic
2. Implement a Linux backend using AT-SPI (Assistive Technology Service Provider Interface) for accessibility
3. Add X11/Wayland screen capture support
4. Add desktop fallback implementations for Windows/Linux (no-op or alternative)

## Acceptance Criteria

- [ ] Platform abstraction layer allows pluggable backends
- [ ] Linux backend can perform basic screen capture and element inspection
- [ ] Desktop computer-use components have platform guards with graceful fallbacks
- [ ] Tests run on Linux without skipping computer-use tests

## Labels

enhancement, help wanted

Contributor guide

Open the contributing guide

Research direction

Start by reading packages/computer-use/src/select-backend.ts and packages/computer-use/src/maka-cu-backend.ts, then inspect the seven files under apps/desktop/src/main/computer-use/. Compare the current macOS-only paths with the acceptance criteria; done means a pluggable Linux backend, basic screen capture and element inspection, guarded desktop fallbacks, and Linux tests that no longer skip computer-use coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
electron, linux, typescript
Domain
accessibility, desktop, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.