Feature Proposal: Browser Control for Gemini CLI

# Feature Proposal: Browser Control for Gemini CLI

## Summary

Enable Gemini CLI to control a web browser through a hybrid architecture combining a **Semantic Agent** (Accessibility Tree-based) with a **Visual Agent** (screenshot-based Computer Use model). This approach optimizes for cost and speed while maintaining the ability to "see" the screen when necessary.

## Motivation

Web automation is increasingly expected in developer tooling. Users need to:

- Debug web applications by inspecting live browser state
- Automate repetitive browser tasks (form filling, testing)
- Interact with web services that lack APIs
- Scrape and analyze web content

A hybrid architecture allows efficient handling of most tasks via the fast, low-cost Accessibility Tree while delegating visually-dependent tasks (color recognition, spatial layout) to multimodal models when needed.

## Proposed Architecture

```mermaid
graph TD
A[User Request] --> B[Semantic Agent]
B --> C{Can I do this via AX Tree?}
C -- Yes --> D[Execute Semantic Tools]
D --> E[Navigate / Click / Type]
E --> B
C -- Need Visuals --> F[Delegate to Visual Sub-Loop]
F --> G[Take Screenshot]
G --> H[Visual Sub-Loop]
H --> I[Execute Visual Tools]
I --> J[Click At / Drag / Scroll]
J --> H
H -- Task Done --> K[Return Result to Semantic Agent]
K --> B
```

### Semantic Agent (Main Loop)

Uses the Chrome DevTools MCP Server to interact with pages via the **Accessibility Tree (AX Tree)**:

```
uid=1_0 RootWebArea "Example Site"
uid=1_1 banner
uid=1_2 link "Home"
uid=1_3 main
uid=1_4 heading "Welcome" level=1
uid=1_5 button "Get Started"
uid=1_6 textbox "Email"
```

Tools: `navigate`, `click`, `fill`, `hover`, `scroll`, `press_key`, `take_snapshot`

### Visual Agent (Sub-Loop)

When the Semantic Agent cannot accomplish a task (e.g., "click the blue button"), it delegates to a Visual Agent powered by the Gemini Computer Use model:

1. Capture screenshot
2. Send to Computer Use model
3. Execute coordinate-based actions (`click_at(x, y)`)
4. Repeat until sub-task complete
5. Return control to Semantic Agent

### Cost Efficiency

| Approach | Input Cost | Output Cost |
| ------------------- | -------------- | ----------- |
| Semantic (AX Tree) | ~1,000 tokens | Low |
| Visual (Screenshot) | ~5,000+ tokens | High |

Converting semantic tasks to visual increases cost per step by ~12x. The Semantic Agent acts as a "cost guardrail."

## Session Management

Two modes via `browserAgentSettings.sessionMode`:

### Existing Mode (`sessionMode: "existing"`)

Attach to user's running Chrome via CDP (requires Chrome M144+):

- Inherits all cookies, logged-in sessions, and state
- Chrome displays permission dialog on first connect
- Banner indicates automation is active
- One-time setup via `chrome://inspect#remote-debugging`

### Isolated Mode (`sessionMode: "isolated"`)

Launch dedicated browser instance:

- Clean automation environment
- Optional persistent profile via `profilePath`
- Works with any Chrome version
- Ideal for CI/CD and reproducible automation

| Aspect | Existing | Isolated |
| ------------------- | ------------------------------ | ------------------------------ |
| Session Inheritance | ✅ Full | ❌ Requires re-login |
| Browser Isolation | ❌ Shares user's browser | ✅ Dedicated instance |
| Chrome Version | M144+ | Any |
| Best For | Live debugging, logged-in apps | CI/CD, reproducible automation |

## Security Considerations

Browser agents expand the attack surface. Built-in guardrails:

- **Navigation restrictions**: Optional domain allowlist
- **Sensitive action confirmation**: User approval for form submissions, file uploads
- **Rate limiting**: Max actions per task to prevent runaway loops
- **Session isolation**: Default to isolated mode

## References

- [Chrome DevTools MCP Blog Post](https://developer.chrome.com/blog/chrome-devtools-mcp-debug-your-browser-session)
- [Gemini Computer Use Model](https://ai.google.dev/gemini-api/docs/computer-use)