Give Agents Eyes and Hands on the Web: Built-in Browser Automation

Summary
Ship a first-party browser tool so agents can open pages, read rendered content, click, fill forms, and take screenshots — without the user wiring up external automation servers.

Component(s) Affected
done
Core / Backend (app, agents, config, providers, utils, local_models)
done
Console (frontend web UI)
not done
Channels (DingTalk, Feishu, QQ, Discord, iMessage, etc.)
done
Skills
not done
CLI
done
Documentation (website)
not done
Tests
not done
CI/CD
not done
Scripts / Deploy
Problem / Motivation
Agents today can fetch raw HTML or call a search API, but the modern web is rendered by JavaScript, gated by logins, and driven by clicks. Anything interactive — checking a price behind a filter, submitting a form, reading a dashboard — is out of reach unless the user finds, installs, and configures a third-party automation MCP server. Built-in tools are also dramatically cheaper: every external-MCP hop adds schema tokens and call overhead, while a native tool is one direct call.

Playwright is already bundled inside the desktop build (_internal/playwright/...) — the engine is literally shipping today with no feature attached to it.

Proposed Solution
Native agent tools: browser_open(url), browser_read() (rendered text/accessibility tree), browser_click(selector/ref), browser_type, browser_screenshot.
Headless by default; optional visible window so users can watch or take over (e.g., to log in once, with the session persisted).
Guardrails through the existing tool_guard: domain allow/deny lists, confirmation policy for form submissions.
Screenshots flow to vision-capable models for "look at this page" reasoning.
Alternatives Considered
External browser-automation MCP servers: heavy setup, extra token cost per call, and another process to babysit.
Fetch-only: fails on any JS-rendered or interactive page.
Additional Context
Playwright's browser driver is already present in the packaged backend — this request is mostly about exposing it as agent tools with a safety layer.

Willing to Contribute
not done
I am willing to open a PR for this feature (after discussion).

Start by inspecting the packaged browser driver under _internal/playwright/... and the existing tool_guard used by the backend. Trace how agent tools are exposed, then define the scope for browser_open, browser_read, browser_click, browser_type, and browser_screenshot; done should include the proposed safety controls, session behavior, and tests, all of which the issue currently marks as incomplete.

Escrito por el modelo de indexación a partir del texto del issue.