openai / openai/codex-plugin-cc

feature request: /codex:computer-use to delegate runtime/UI verification (app launch, screenshots, simulators)

Open
#446 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
33.3k
Forks
2.3k
PR merge metrics
No merged PRs in 30d

Description

Problem

The plugin has two delegation shapes today: reviewing code (/codex:review, /codex:adversarial-review: read-only, git-scoped, structured findings) and changing code (/codex:rescue: write-capable, free-form). Neither answers the question that comes right after an implementation lands: does it actually work at runtime?

Codex has computer-use capability, but the plugin exposes no path to it. Verification is its own job with its own requirements. It needs to be read-only on source, it needs evidence artifacts (screenshots, logs), it needs a pass/fail/blocked verdict, and it often needs GUI-level access. Prompting codex-rescue to "test the app" fights that agent's write-capable defaults and thin-forwarder design, and returns no structured verdict. #354 describes the same category error for review: bending the wrong tool with prompt instructions.

Use cases

  1. Post-implementation UI check. Claude finishes a signup form, then /codex:computer-use verify the signup flow on localhost:3000, happy path plus validation errors --wait. Codex opens the browser, walks the flow, screenshots each step, and returns pass or fail with evidence paths.
  2. Background verification while work continues. Codex verifies feature A with --background (results via the existing /codex:status and /codex:result) while Claude implements feature B.
  3. Environments Claude can't reach: iOS simulator, Electron app, desktop window state. A second agent that didn't write the code doesn't inherit the implementer's assumptions.

Proposed shape

Everything below is a sketch to anchor discussion; the design is yours. The idea is to mirror the review architecture so the existing runtime does most of the work.

  • /codex:computer-use [--wait|--background] [--full-access] [what to verify, how to launch, what counts as pass]
  • A companion subcommand with the same foreground/background execution model as review, with jobs visible to status, result, and cancel.
  • A verification prompt template under prompts/ that requires a self-contained brief: behavior to verify, platform and app type, launch commands and test credentials, whether source edits are allowed (default: no), artifact paths, and the return contract.
  • A structured output schema under schemas/: verdict: pass|fail|blocked, steps_performed[], observed_behavior, screenshots[], findings[], confidence.
  • A runtime-managed artifact directory per job for screenshots, logs, and the report.
  • Possible v2: a model-invocable codex-verifier subagent so Claude can dispatch verification proactively. That intersects #269/#238 and #354, so command-only is a fine v1.

Safety defaults

  • Source edits forbidden by default; an unexpected diff is a finding, not a success.
  • Sandbox tiers: workspace-write for non-GUI runtime checks. GUI automation (app launch, simulators, screenshots) sits behind an explicit opt-in. #145 already requests the same --full-access flag for task; this feature would be its second consumer.
  • Disruption boundary: launching apps, simulators, or browsers to verify the requested work is within the job's mandate. Closing the user's apps, changing system settings, or acting on real accounts or data requires explicit consent in the brief.
  • No auto-fix after a fail verdict. The existing result-handling rule applies unchanged.

Fallbacks

  • Codex missing or unauthenticated: stop and point to /codex:setup, same as every other command.
  • Headless environment, no display, simulator absent: return verdict: blocked with the reason instead of hanging.
  • Long runs (app boots and simulators are slow): background mode is the primary path, and the Bash 10-minute-ceiling work in #370 applies here too.
  • Model or account without computer-use capability: a specific error instead of a generic API failure.

Alternatives considered

  • Prompting codex-rescue: wrong defaults (write-capable), no verdict schema, no artifact contract, and its own guidance forbids verification-shaped wrapper work around the run.
  • Claude-side browser automation (Playwright MCP and similar): useful, but not independent, since the implementer verifies itself. It also occupies Claude's turn instead of running in parallel.
  • A local personal skill: works today, and it is where this idea comes from (see Inspiration). But every user then reinvents prompt contracts, artifact handling, sandbox policy, and the cross-platform process management this plugin's runtime already solves centrally.

Inspiration

I got the idea after watching Theo's (t3.gg) "A proper guide to Fable 5" (https://www.youtube.com/watch?v=8GRmLR__OGQ, from ~23:05), where he shows a hand-rolled codex-computer-use skill next to sibling skills for Codex-delegated review and implementation. His version defines the verification stance, a do-not-use boundary (not for ordinary code reading, typechecking, linting, or tests Claude can run directly), a 4-step workflow around codex exec, and a prompt-requirements checklist. It works as a personal skill, but this capability makes more sense living centrally in the official plugin, where the runtime, job management, and sandbox policy already exist, than as something each user rebuilds for themselves.

Together with #354 (reviewer subagent) and #263 (/codex:implement), this would complete the set: review, implement, verify.

Duplicate check

Searched titles, bodies, and comments across all open and closed issues and PRs for "computer use", "computer-use", screenshot(s), browser automation, simulator, playwright, "ui test", "visual verification", "verify ui". No existing request; all hits were incidental. Closest in spirit are the image-generation PRs #357/#271, which are a different capability.

Open questions

  • Platform phasing: is macOS/Linux-first acceptable, given the Windows process/spawn complexity already in the tracker?
  • Screenshot artifact retention: prune with job state, keep the last N, or leave it to the user?
  • Naming: /codex:computer-use (mechanism, matches OpenAI terminology) vs /codex:verify (intent, matches the review/rescue verb family). No strong attachment.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing review command and its foreground/background job flow, including /codex:status, /codex:result, and /codex:cancel. Then inspect the prompts/ and schemas/ areas and the runtime artifact handling described in the issue. Done means a command-only verification path with explicit safety defaults, structured pass/fail/blocked results, evidence artifacts, and the documented fallback behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
cli, developer-experience, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.