Hmbown / Hmbown/Codewhale

Shared-machine coexistence: run all day alongside an active human

Open
#6,301 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
41k
Forks
3.6k
Avg merge
13h 59m
Merged PRs (30d)
299

Description

Operating Computer Use on the same Mac a human is actively using is disruptive today (observed during the 0.8.0 muse acceptance pass, Hmbown/Codewhale#6300): agent windows appear on the human's screen, apps launch visibly, every window-record key takes a front-process lease that can swallow or misdirect concurrent human keystrokes, and one background re-bind observed the scratch app frontmost mid-session. Observation (AX queries, screenshots, menus) is invisible; input leases and visible artifacts are the cost.

Gaps, grouped:

1. Hide the work: launch hidden, keep agent windows off the human's Space / ordered back; clean up everything created. Ceiling note: macOS has one front process per user session, so a second display does not isolate input. Sustained autonomous work belongs on a separate computer (VM / cloud / second Mac via the existing registry); same-screen is for brief idle-gated assists.
2. Yield to the human: idle-gating (lease only when idle N seconds), yield-on-input (abort lease on HID activity), dispatch-time frontmost revalidation, lease-free routes (AX actions) always preferred over window-record.
3. Honest interference accounting: receipts report lease duration plus whether user input occurred during the lease; per-day audit log of what the agent touched, per app.
4. Human controls: global working indicator with one-click pause, quiet hours, per-app hands-off list. Make "do not work in apps the user is actively editing" system-enforced (frontmost / recent-input signals with refuse-or-escalate), not model advice.
5. Contention correctness: agent transfers via a private pasteboard with human-clipboard restore + verify; no stale-input replay after pause; safe no-op when the human switches windows between observe and act.
6. Scheduling and manners: batch disruptive actions; reads anytime, writes when idle; rate-limit leases.

Acceptance sketch: a human types continuously in their own apps while the agent runs an hour-long task in its own apps — zero lost or misdirected keystrokes, zero focus thefts, zero agent windows on the human's Space; the audit log shows every lease with duration and zero user-input-during-lease flags. Pause and the hands-off list are honored in flight (<1s).

Related: Hmbown/Codewhale#6300 (acceptance pass where this was observed), Hmbown/Codewhale#5927 (wrong-target input incident class), Hmbown/Codewhale#5856 (Engine acceptance incl. no stale replay).

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by mapping the existing Computer Use window, input-lease, pasteboard, pause, and audit-log paths, then review the related issues for constraints and prior incidents. Done means the stated acceptance sketch passes: no lost input or focus theft, no windows on the human’s Space, accurate lease receipts, and pause and hands-off controls honored in flight.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, rust
Domain
desktop, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.