microsoft / microsoft/GitHub-Copilot-for-Azure

Replace microsoft-foundry skill toolbox MCP smoke test with a script

Open
#2,542 1 comment 0 reactions 1 assignee Claimed by @tmeschter View on GitHub
microsoft-foundry skills untriaged
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Summary

Copilot has identified a portion of a skill that is a good candidate for replacement with a script.

The candidate is the **toolbox MCP endpoint smoke test** in the `microsoft-foundry` skill (`foundry-agent/create/references/`) — a token → initialize → tools/list → tools/call JSON-RPC handshake documented twice (bash and PowerShell).

## Candidate description

To verify a toolbox MCP endpoint, the skill runs a fixed four-step handshake:

1. `az account get-access-token --resource https://ai.azure.com`.
2. POST `initialize` (threading the returned `mcp-session-id`).
3. POST `tools/list`.
4. Optional POST `tools/call`.

Every request carries the same constant headers (`Authorization: Bearer`, `Content-Type`, `Foundry-Features: Toolboxes=V1Preview`) and the `?api-version=v1` requirement, and the docs reduce the large JSON-RPC responses to a checklist (`result.tools[]` non-empty; each tool has `name`/`description`/`inputSchema`; `server_label` prefix present).

This is a strong script candidate because it is:
- **A fixed, branch-free sequence** with constant headers and a session-id thread.
- **Output-heavy → checklist** — collapses ~50 lines of raw curl/JSON into a tool-name list plus pass/fail.
- **Duplicated across two files** — `toolbox-reference.md` documents it as bash/curl (initialize + tools/list + tools/call); `use-toolbox-in-hosted-agent.md` documents the same `tools/list` as a PowerShell `Invoke-RestMethod`. One bash + one PowerShell script covers both.

**Sketch — `test-toolbox-endpoint.{sh,ps1}`:**
- **Input:** `--project-endpoint`, `--toolbox-name`, optional `--tool` + `--args` for the call step.
- **Output:** the discovered tool-name list and a pass/fail on the checklist.

> Only the optional `tools/call` step needs a tool name + sample arguments (judgment). The token / initialize / list portion is fully mechanical.

## Affected file and lines

- [`foundry-agent/create/references/toolbox-reference.md` — testing the toolbox endpoint (L124–L175)](https://github.com/microsoft/GitHub-Copilot-for-Azure/blob/3890cbfb65c548ce8daa96cabd1d8de63f7bbcca/plugin/skills/microsoft-foundry/foundry-agent/create/references/toolbox-reference.md#L124-L175)
- [`foundry-agent/create/references/use-toolbox-in-hosted-agent.md` — end-to-end smoke test (L212–L228)](https://github.com/microsoft/GitHub-Copilot-for-Azure/blob/3890cbfb65c548ce8daa96cabd1d8de63f7bbcca/plugin/skills/microsoft-foundry/foundry-agent/create/references/use-toolbox-in-hosted-agent.md#L212-L228)

## Next steps

1. **Evaluate the candidate** — confirm the steps are stable and parameterizable, and that the script captures everything the skill needs.
2. **Create both a bash _and_ a PowerShell version** of the script so the skill works across platforms.
3. **Run integration tests** to verify the scripts behave correctly and the skill still completes end-to-end.

## Background Information

### Why replace regular steps with scripts

Replacing a regular, well-defined series of steps with a script can:

- **Reduce token usage** — the skill no longer needs to spell out each command and parse large command output inline; the agent invokes one script and reads a compact result.
- **Improve reliability** — the logic is written and tested once, instead of being re-derived by the agent on every run.
- **Improve determinism** — the same inputs always produce the same steps and output, removing run-to-run variation.
- **Improve speed of execution** — a single script call replaces multiple round-trips of command generation, execution, and large-output parsing.

### Authoring notes for the scripts

- **Reference scripts with markdown links**, not just a bare path to the script file.
- **Include examples** in the skill showing how to run each script (sample invocation with arguments).
- **Briefly explain what each script does** where it is referenced.
- **The script output should explain what it did**, so the agent and user can understand the result without re-inspecting raw command output.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.