MCP Debug Mode: Expose UI State and Widget Tree
- Dominant language
- Dart
- Stars
- 179k
- Forks
- 31.1k
- PR merge metrics
- PR metrics pending
Description
### Use case
Flutter has no built-in way for AI tools or external developer agents to understand what the app looks like at runtime. When the app runs in debug mode, AI-assisted tools cannot access a screenshot, widget tree, or layout information. This makes it difficult for AI to help with UI debugging, layout analysis, and automatic test generation.
I am requesting a feature that allows Flutter to expose runtime UI state to AI tools or debugging tools.
Problem
AI agents currently only have access to source code — not the live UI.
They cannot:
See what the screen looks like
Inspect the widget hierarchy
Analyze layout constraints or bounds
This limits AI-assisted development that relies on understanding the actual rendered UI.
Alternatives considered
Screenshot packages exist, but they don't expose a full widget tree or structured data.
Devtools inspector is visual-only and not accessible programmatically.
No package on pub.dev provides a standardized protocol like MCP for structured tool communication.
There is no existing solution that exposes the UI in a safe, standardized, debug-only way.
### Proposal
Allow Flutter apps (in debug mode only) to run a lightweight MCP (Model Context Protocol) server. This would let tools request structured information about the current UI.
What Flutter should support
`getScreenshot` → return current screen pixel buffer
`getWidgetTree` → return the rendered widget/element tree
`getLayoutInfo` → return bounding boxes and constraints
This would allow AI agents and development tools to understand what is currently rendered on screen.
Why MCP
MCP is a simple, standardized protocol designed for safe tool-to-application communication.
Using it avoids building custom RPC systems and enables compatibility with modern AI tools.
Should Flutter provide this directly?
Yes — because:
Flutter already has access to the widget tree and render tree
Screenshot and inspector logic exist internally
A framework-level implementation is more reliable and consistent than a third-party package
A package could attempt this, but it would be much more limited.
Contributor guide
Assessment
This issue has not been assessed yet.