Add an interactive "human-assist" mode for GUI/emulator actions in Codex

What variant of Codex are you using?

App, CLI

What feature would you like to see?

Summary

When Codex is performing actions in a GUI context it doesn't have reliable visual/coordinate access to (e.g., an Android emulator, a desktop app, a browser window opened during a task), it currently has to either (a) burn time and tokens inferring element positions and attempting the action itself, or (b) stop and ask the user to perform the action and report back manually. Both are worse than they need to be. This issue proposes a lightweight, opt-in "assist mode" that lets the user perform the physical action while Codex keeps driving the task.

Motivation

The underlying capability already exists, Codex can prompt the user for input/approval mid-session (e.g., approval mode for shell commands) and interpret simple responses. What's missing is a structured handoff mechanism for GUI actions specifically, so the human can act as Codex's "hands" for the parts it can't reliably do itself, without derailing the session into manual back-and-forth or requiring the user to fully take over.

Example use case

I asked Codex to test an Android app in an emulator it launched from the CLI. After starting the emulator, Codex had no reliable way to identify exact button coordinates, so it spent significant time and tokens guessing, clicking, and re-checking screenshots to find the right UI element.

Instead, the flow could be:

1. After launching the emulator (or entering any GUI-driven step), Codex asks a single yes/no question: "Would you like to assist with GUI interactions for this task?"
2. If "No": Codex proceeds exactly as it does today, working out coordinates/actions on its own.
3. If "Yes": whenever Codex needs to take a GUI action, instead of attempting it blindly, it presents a small structured prompt describing the action it wants performed, e.g.:
   • "Please tap the 'Login' button in the top-right corner."
   • Response options: Done (confirms the action was taken) or a free-text field for "I don't understand / clarify" if the instruction is unclear.
4. Codex waits for the response, then continues the task using that confirmation as ground truth, instead of re-verifying via screenshots or guesswork.

Proposed solution

• Add a toggleable "assist mode" that can be offered at the start of any GUI-interaction-heavy task (emulator testing, desktop app automation, browser-driven tasks, etc.), similar in spirit to Codex's existing approval-mode prompts for shell commands.
• When enabled, replace autonomous GUI action attempts with a simple confirm/clarify prompt per action, surfaced through the CLI (or IDE extension, where applicable).
• When disabled (or declined), fall back to current autonomous behavior.
• Should apply across Codex's supported surfaces (CLI and any IDE/editor integrations) since GUI/emulator-driven tasks can originate from either.

Expected benefit

• Saves tokens and time on tasks where Codex has to laboriously infer UI coordinates or state.
• Increases reliability, since a human confirms the action instead of Codex assuming success.
• Keeps the human in the loop without forcing them to fully take over the task or narrate results back manually.
• Fits naturally alongside existing approval/confirmation UX patterns already used for command execution.

Additional information

No response

Start by tracing the existing approval-mode prompts in the CLI and the IDE/editor integrations mentioned in the issue. Define how an opt-in assist mode would be offered for GUI-heavy tasks and how the Done or clarification responses flow back into the session. Done means the behavior is consistent across supported surfaces while declined assistance preserves autonomous behavior.