flutter / flutter/flutter

MCP Debug Mode: Expose UI State and Widget Tree

Open
#178,750 6 comments 0 reactions 0 assignees View on GitHub
c: new feature c: proposal engine P3 team-engine triaged-engine
Dominant language
Dart
Stars
179k
Forks
31.1k
PR merge metrics
PR metrics pending

Description

### Use case

Flutter has no built-in way for AI tools or external developer agents to understand what the app looks like at runtime. When the app runs in debug mode, AI-assisted tools cannot access a screenshot, widget tree, or layout information. This makes it difficult for AI to help with UI debugging, layout analysis, and automatic test generation.

I am requesting a feature that allows Flutter to expose runtime UI state to AI tools or debugging tools.

Problem

AI agents currently only have access to source code — not the live UI.
They cannot:

See what the screen looks like

Inspect the widget hierarchy

Analyze layout constraints or bounds

This limits AI-assisted development that relies on understanding the actual rendered UI.

Alternatives considered

Screenshot packages exist, but they don't expose a full widget tree or structured data.

Devtools inspector is visual-only and not accessible programmatically.

No package on pub.dev provides a standardized protocol like MCP for structured tool communication.

There is no existing solution that exposes the UI in a safe, standardized, debug-only way.

### Proposal

Allow Flutter apps (in debug mode only) to run a lightweight MCP (Model Context Protocol) server. This would let tools request structured information about the current UI.

What Flutter should support

`getScreenshot` → return current screen pixel buffer

`getWidgetTree` → return the rendered widget/element tree

`getLayoutInfo` → return bounding boxes and constraints

This would allow AI agents and development tools to understand what is currently rendered on screen.

Why MCP

MCP is a simple, standardized protocol designed for safe tool-to-application communication.
Using it avoids building custom RPC systems and enables compatibility with modern AI tools.

Should Flutter provide this directly?

Yes — because:

Flutter already has access to the widget tree and render tree

Screenshot and inspector logic exist internally

A framework-level implementation is more reliable and consistent than a third-party package

A package could attempt this, but it would be much more limited.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.