Azure / Azure/azure-rest-api-specs

ARM API Reviewer agent: exceeds the 30,000 character prompt limit and cannot be selected in the browser

Open
#45,843 0 comments 0 reactions 0 assignees View on GitHub
ARM API Reviewer agent
Dominant language
TypeSpec
Stars
3.1k
Forks
5.9k
Avg merge
2d 22h
Merged PRs (30d)
444

Description

### Summary

The **ARM API Reviewer** custom agent cannot be selected in the GitHub.com agent picker. At `https://github.com/Azure/azure-rest-api-specs/agents` the entry is greyed out with:

```
Invalid config: "prompt" exceeds max length of 30000
```

Sibling agents in the same directory load normally, so the repository wiring is fine. The agent definition is simply far too large for that surface.

### Measurements

| File | Characters | Status |
| --- | --- | --- |
| `.github/agents/arm-api-reviewer.agent.md` | 181,156 | 6x over the limit, rejected |
| `.github/agents/data-plane-api-reviewer.agent.md` | 8,612 | loads |
| `.github/agents/hero-scenarios.agent.md` | 382 | loads |
| `.github/agents/arm-api-review-critic.agent.md` | 68,113 | not shown, but by design (see below) |

Where the size sits in the ARM reviewer. Note the split below counts `##` headings inside fenced templates, so the report and preamble templates actually belong to Review Workflow:

| Area | Characters |
| --- | --- |
| Review Workflow, Steps 1-10 including templates | ~124,900 |
| Failure Modes & Recovery | 14,369 |
| Pre-Presentation Invariant | 13,178 |
| Constraints | 5,199 |
| Persona, repos, scope, glossary, mode | ~23,500 |

### What the limit actually measures

Per [custom agents configuration](https://docs.github.com/en/copilot/reference/custom-agents-configuration) and [creating custom agents](https://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-cloud-agent/create-custom-agents):

> Define the agent's behavior, expertise, and instructions in the Markdown content below the YAML frontmatter. The prompt can be a maximum of 30,000 characters.

Key findings from that documentation:

- It measures the **Markdown body below the frontmatter**, in **characters**.
- It is a **github.com cloud-agent constraint**. The [VS Code custom agents docs](https://code.visualstudio.com/docs/agent-customization/custom-agents) state no equivalent limit, which is why the agent works in VS Code today and fails only in the browser.
- There is **no documented per-repo or per-org setting** that raises it.
- `.agent.md` has **no `imports:` mechanism**. The documented property list is `name`, `description`, `target`, `tools`, `model`, `disable-model-invocation`, `user-invocable`, `infer`, `mcp-servers`, `metadata`. Content referenced by path in prose and read at runtime through the agent's tools does **not** count against the limit.
- `arm-api-review-critic.agent.md` is absent from the picker because it sets `user-invocable: false`, not because of its size. No action needed there.

### Suggested approach

Follow the pattern already proven in this repository by `data-plane-api-reviewer.agent.md`: a thin contract plus a `## References` section of relative links, with explicit "read only what you need" and "do not load X" guidance, leaving detail in `.github/skills/azure-api-review/references/` and `.github/agents/protocols/`.

Applied here, that means moving the Review Workflow, the report and preamble templates, and Failure Modes into referenced files, while keeping the **Pre-Presentation Invariant** inline because it is the load-bearing gate.

### Risk

This is a behavioral change, not just a mechanical move. Rules relocated from the prompt into "read this file" instructions are weaker if the agent skips the read. The eval suite under `.github/skills/evals/arm-api-reviewer/` is the control, and a full run should be compared before and after.

### Acceptance criteria

- `.github/agents/arm-api-reviewer.agent.md` body is under 30,000 characters and the agent is selectable at `https://github.com/Azure/azure-rest-api-specs/agents`.
- No review rule is lost; relocated content is reachable through explicit references.
- The eval suite passes at parity with the pre-change baseline.
- The agent continues to work unchanged in VS Code.

### References

- Related: #45841, #45013, #44356

Contributor guide

Open the contributing guide

Research direction

Read .github/agents/arm-api-reviewer.agent.md and compare its structure with .github/agents/data-plane-api-reviewer.agent.md, then inspect the referenced files under .github/skills/azure-api-review/references/ and .github/agents/protocols/. Run the eval suite under .github/skills/evals/arm-api-reviewer/ against the pre-change baseline. Done means the body is under 30,000 characters, the browser agent picker accepts it, all rules remain reachable, evals are at parity, and VS Code behavior is unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, markdown
Domain
devtools, tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.