decentraland / decentraland/unity-explorer

Explorer MCP server V2

Open
#9,393 3 comments 0 reactions 2 assignees Claimed by @popuz View on GitHub
feature sdk
Dominant language
C#
Stars
23
Forks
17
Avg merge
2d 19h
Merged PRs (30d)
107

Description

## Improvements + Features for V2

### Updates to [SDK-SKILL](https://github.com/decentraland/sdk-skills/tree/main/unity-explorer-mcp)

- Make it use `use --skip-auth-screen true` when the agent opens the Explorer itself.

### Enable using input not bound to entities

The agent cannot use non-pointer-event input throught MCP.

Image

Related to https://github.com/decentraland/unity-explorer/issues/9393#issuecomment-5145548031

### Enable seeing pointer events HOVER FEEDBACK

Currently the agent cannot see it, only click directly

### Enable clicking screen-space UI

Currently not possible.

### Mechanism to be able to cancel an in-flight tool

**Proposed implementation**
```
Add real per-command cancellation without breaking the stateless "one POST = one message" model:

1. On `tools/call`, create a linked `CancellationTokenSource`, register it in a `Dictionary`, and pass its token (not the shared server token) to the tool.
2. Implement a `notifications/cancelled` handler (its `params` carry `requestId`) — look up the CTS by id and cancel it.
3. Clean up the dictionary entry on call completion (`finally`).
```

### Enable multiplayer local testing for agents

Maybe the easiest way would be making the Agent use `metaforge` CLI tool to handle the accounts setup

### (For debugging) CRDT steate get + CRDT history get

Tools for allowing the agent check the CRDT state of the scene + the full CRDT commands history

Also flag WHO sent that CRDT command that affected the state of the scene.

Related draft PR as technical reference: https://github.com/decentraland/unity-explorer/pull/9547

### (For debugging) Selective toggling of tools

Can we disable some tools? (with a paremter, to separate tools for creators and for debugging)
- We can make `--mcp` arg with parameters. Default profile would be creator, and for devs one need to specify `--mcp dev`. Then it is just an `if` in the `McpServerPlugin.InjectToWorld` where it assembles `McpToolRegistry`. Different profiles will have just differently assembled registry

### (For debugging) Connect MCP server to CDP Protocol - get network logs
- Connect to CDP protocol ([reference 1](https://github.com/decentraland/chrome-devtool-protocol-unity/pull/5), [ref 2](https://github.com/decentraland/unity-explorer/commit/72f93ce78755b608b45eeb60be38c1d0aa565e96)) -> maybe the MCP server can get the network info that CDP gets.
- Potential approach already implemented at PR https://github.com/decentraland/unity-explorer/pull/9437

### (For debugging) Get Unity player/editor logs (beyond `get_scene_tools`)
Potential approach already implemented at PR https://github.com/decentraland/unity-explorer/pull/9436

### Enable scene metrics analisys

Once we get in scene graphics metrics (triangles, textures, and materials count) we should provide a tool to fetch and let the agent suggest optimization areas according to those values

### Improve `get_entity_details` + `list_scene_entities`

Currently when the agent wants to find out something about a specific Entity it does the following:
1. Calls `list_scene_entities` tool and gets a collection of (arch ecs) entities
2. Calls `get_entity_details` tool for each entity in the collection until it finds the one it's looking for

We could empower `list_scene_entities` to return more info on each entity instead of only the entity ID, e.g.
Image

Additionally, if we implement @popuz 's idea of allowing tools to have "short | full" modes, we could add the expanded data only when the `list_scene_entities` tool is called as `full`

Image

### Missing atmospheric blend effect in screenshots

Apparently worldOnly screenshot path renders the skybox's sun/space backdrop layer raw — without the atmospheric blend the on-screen pipeline applies

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.