Automattic / Automattic/studio

Studio Code: compose prompts and tools from resolved site capabilities

Open
#3,333 2 comments 0 reactions 0 assignees View on GitHub
Enhancement YOLO
Dominant language
TypeScript
Stars
517
Forks
95
Avg merge
1d 22h
Merged PRs (30d)
162

Description

## Problem

Studio Code currently asks the model to discover WordPress.com site plan context inside the agent loop. In remote-site mode, the system prompt instructs the agent to call `GET /` before doing any work, read `plan.product_slug`, and then infer what is allowed from static prompt prose.

That makes the prompt responsible for capability discovery, capability interpretation, tool affordance shaping, and user-facing explanation. It also makes plan/capability drift easy: the model sees broad tools and endpoint guidance even when the active site cannot use those capabilities.

## Current behavior observed

- `apps/cli/ai/system-prompt.ts` tells the agent to call `GET /` with `apiNamespace: ""` before work and check `plan.product_slug`.
- `apps/cli/ai/system-prompt.ts` hardcodes remote plan/design guidance and lists broad WP.com endpoint families, including plugin/theme endpoints.
- `apps/cli/ai/agent.ts` builds remote MCP tools with `createRemoteSiteTools( wpcomAccessToken, activeSite.wpcomSiteId! )`; no site capability object is passed.
- `apps/cli/ai/tools.ts` exposes remote tools via `createRemoteSiteTools( token, siteId )`, which always includes the same generic `wpcom_request` plus site/screenshot/pull tools.
- `apps/cli/ai/wpcom-tools.ts` strips `plan.features` from the `GET /` response and comments that the system prompt hardcodes what each plan tier can do.
- Remote `SiteInfo` / `site.selected` events carry `name`, `url`, and `wpcomSiteId`, but not plan/capability metadata.
- Sync code elsewhere already derives at least one capability dynamically (`studio-sync`) from `site.plan.features.active`, but that normalized information is not threaded into the agent runtime.

## Proposal

Resolve the active site capabilities before starting the model turn, then compose the prompt, tool descriptions, tool schemas, and runtime gates from that same normalized capability object.

Suggested flow:

```text
User sends message

Studio resolves active site/environment

Studio fetches or refreshes site capability snapshot

Studio builds capability-aware MCP tools and schemas

Studio builds a smaller capability-aware system prompt

Agent starts with a scoped operating environment
```

The model should not need to call `GET /` as a first step just to know what it is allowed to do.

## Suggested capability shape

```ts
type WpcomSiteCapabilities = {
planSlug: string;
planName: string;
isFree: boolean;
isExpired: boolean;
canEditContent: boolean;
canUseCustomCss: boolean;
canInstallPlugins: boolean;
canUploadThemes: boolean;
canRunCustomPhp: boolean;
canUseStudioSync: boolean;
};
```

This should be treated as site-specific, not user-global. The same user can manage multiple WordPress.com sites with different plans/capabilities.

## Desired architecture

- Prompt: summarize current site capabilities and policy, not every endpoint/tool.
- Tool descriptions: describe only actions relevant to the active site capabilities.
- Tool schemas: constrain input where possible, e.g. allowed sync options for the current site.
- Runtime gates: enforce the same capabilities even if the model attempts an unsupported operation.

Example for a Premium/Explorer site:

```text
Available:
- Edit posts/pages/media
- Update supported block content
- Use custom CSS

Unavailable:
- Install plugins
- Upload custom themes
- Run custom PHP
- Push plugin/theme code
```

`site_push` should either omit unsupported options from its schema/description or reject them before execution with a precise capability error.

## Why this matters

- Removes brittle “first call GET /” prompt instructions.
- Reduces prompt size and duplicated tool/endpoint lists.
- Prevents unavailable capabilities from being advertised to the model.
- Keeps capability truth in code instead of prose.
- Makes user-facing refusals more precise and product-aligned.

## Notes

This is separate from any tactical prompt copy correction. The goal is to make Studio Code capability-aware before the model loop starts, then expose a scoped prompt and tool surface for the active site.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.