Add browser-based automation tools to the core catalog
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 152
- Forks
- 16
- Avg merge
- 14h 48m
- Merged PRs (30d)
- 536
Description
Summary
mecatl has no way to drive a real browser. The web surface today is two tools: WebSearch (discover sources, engine/adapter/search/websearch.go) and the built-in WebFetch (engine/adapter/webfetch, re-exported from internal/adapter/tools). Per ADR 0105, WebFetch is deliberately bounded: one absolute HTTP(S) URL, GET only, no headers/cookies/credentials, HTML parsed via x/net/html without rendering, script execution, or subresource loading. JavaScript-rendered sites are called out as intentionally unsupported.
That leaves a real gap. An agent cannot:
- Read a page whose content only renders client-side (SPAs, JS-gated docs).
- Click, fill forms, or complete multi-step / authenticated flows.
- Wait on selectors/navigation, or execute page JS.
- Capture screenshots for visual verification.
This is a request to add first-class browser automation to the tool catalog.
Motivation
A growing class of agent tasks need a rendered, interactive browser — QA of web UIs (including our own mecatl-studio surface), scraping client-rendered content, and driving multi-step flows. Bounded WebFetch correctly refuses these; there is no fallback short of an ad-hoc MCP server or shelling out, which sidesteps the catalog's permission model and the FenceUntrusted guarantees.
Proposed capability
A set of browser tools (minimum viable surface):
- Navigate to a URL
- Extract rendered (post-JS) DOM / text
- Click element, fill + submit form
- Wait for selector / navigation
- Screenshot (artifact handling TBD)
- Optionally: evaluate JS in page context
Fit with existing architecture
- Where it lives. WebFetch/WebSearch are engine reference adapters re-exported through
internal/adapter/tools. If browser automation ships in-tree it likely follows the same shape (engine/adapter/browser+ re-export), joining the catalog viatools.All()/Register()ininternal/adapter/tools/tools.go. But a headless-browser dependency (Playwright/Chromium) is far heavier thanx/net/html, so an MCP-server integration (e.g. Playwright MCP behind the existing MCP seam) may be the better first cut — this is the key design decision to settle. See open questions. ReadOnly()dispatch. Read-style ops (navigate, extract, screenshot) are read-parallel; interaction ops (click, fill, submit, evaluate JS) mutate remote state and must reportReadOnly() == falseso the agent loop dispatches them serially (gauntlet #4).- Untrusted output. Rendered page text is untrusted and prompt-injection-prone — same trust boundary WebFetch/WebSearch handle. It must go through the
agent.FenceUntrustedchoke point rather than re-implementing framing in a new adapter. - Session profiles. Decide inclusion in
All()vsNoFS(). A browser tool needs no workspace, so likeWebFetch/FetchMcpResourceit could ride in the no-fs profile — but its weight and blast radius argue for it being opt-in / off by default.
Safety considerations (must not regress ADR 0105 posture)
WebFetch closes redirect SSRF and DNS-rebinding paths and refuses private/loopback/link-local/metadata targets. A real browser reopens all of that plus more (arbitrary subresource loads, JS execution, cookie/credential handling, downloads). Requirements to work out before shipping:
- SSRF / egress controls at least as strong as
WebFetch— no navigation to private/metadata/loopback ranges; re-validate on in-page navigations and redirects. - Sandboxing / isolation for the browser process; bounded resource use (time, memory, page/tab count).
- Explicit posture on credentials, cookies, and file downloads (default deny).
- Default-off / permission-gated, consistent with the guardrails model (ADR 0021).
- Screenshot/artifact storage and size caps; model-visible output bounded like the 25 KB WebFetch cap.
Open questions
- Native in-tree adapter vs. MCP server (Playwright MCP)? Weigh dependency weight, cross-platform browser provisioning, and maintenance vs. the tighter integration/guarantees of an in-tree adapter. (Recommend evaluating MCP-first.)
- Headless vs. headed; where do Chromium/Playwright binaries come from in deploy images?
- Which session profiles include it, and is it default-off?
- Screenshot artifact handling over the gRPC + HTTP/SSE API surface.
- Do we need a dedicated ADR (as WebFetch got 0105) before implementation?
Scope / not-in-scope
- In scope: the tool surface above, its safety model, and catalog/permission wiring.
- Out of scope (for now): full crawler orchestration, video capture, browser-extension automation.
Filed on behalf of @Derek2Tu. Grounded against main: ADR 0105, internal/adapter/tools/tools.go, engine/adapter/webfetch, engine/adapter/search/websearch.go.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ADR 0105, internal/adapter/tools/tools.go, engine/adapter/webfetch, and engine/adapter/search/websearch.go to understand existing adapters, catalog profiles, and trust boundaries. Resolve the native adapter versus MCP design and the safety, session, and artifact questions before implementation. Done means an agreed browser-tool surface and documented permission and isolation model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, security, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100