dotnet / dotnet/aspnetcore

Blazor automation feature suggestion

Open
#66,287 2 comments 1 reaction 0 assignees View on GitHub
area-blazor feature-request
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 6h
Merged PRs (30d)
290

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Is your feature request related to a problem? Please describe the problem.

Like everyone else I'm developing using AI agents. I'm using tools like Playwright MCP to allow the agent to run and debug the application by itself. It works somewhat, but often UI frameworks use HTML that are not easy for the agents to use correctly. The DOM is a strange hybrid between semantic/aria and graphical/rendering which can be hard to use correctly for agents. For example using overlays for dropdowns/menus etc.

### Describe the solution you'd like

Actually i have two separate suggestions:

**Suggestion 1:**
The first suggestion is that Blazor allows inspection of the the component render tree, so we can build custom "snapshots" for agents to inspect. This **may** be possible with "internal only" classes, but I have not successfully made that work.

**Suggestion 2:**
The suggestion i like the most is that Blazor provides a standard "automation framework" that allows authors of UI libraries to enable automation for their components.

I have build my own custom library for this, but that requires me to wrap every single third-party component I'm using.
It would be much better if this was a standard everyone could follow.

This is an extremely simple button implementation using my `AutomationComponent` to enable automation interaction:
```

@Label

@code {

[Parameter]
public string? AutomationId { get; set; }

[Parameter, EditorRequired]
public string Label { get; set; } = null!;

[Parameter]
public bool Enabled { get; set; } = true;

[Parameter]
public EventCallback OnClick { get; set; }

private Task HandleClickAsync()
{
if (!Enabled) throw new InvalidOperationException("Button is disabled.");
return OnClick.InvokeAsync();
}

private async Task HandleActionAsync(string actionName, JsonElement[] args)
{
if (Enabled && actionName == "Click")
{
await HandleClickAsync();
return null;
}
throw new NotSupportedException($"Action '{actionName}' is not supported.");
}
}
```

By using cascading values the components are build into a tree that can be exposed to the agent.

When enabled the automation library creates a global "window.automation" JavaScript object that allows getting a snapshot of all components on the page and also allow calling actions on them.

A snapshot looks something like this:
```
{
"components": [
{
"id": "home-page",
"type": "page",
"description": "The home page of the application",
"state": {
"clickCount": 0
},
"children": [
{
"id": "home-page/button-1",
"type": "button",
"description": "Click me 1",
"state": {
"enabled": true
},
"actions": [
{
"name": "Click",
"description": "Clicks the button"
}
]
},
{
"id": "home-page/button-2",
"type": "button",
"description": "Click me 2",
"state": {
"enabled": false
},
"actions": [
{
"name": "Click",
"description": "Clicks the button"
}
]
}
]
}
],
"links": [
{
"url": "https://learn.microsoft.com/aspnet/core/blazor",
"description": "Blazor documentation on Microsoft Learn"
}
]
}
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.