microsoft / microsoft/GitHub-Copilot-for-Azure

Replace microsoft-foundry skill fine-tuned model deploy-and-verify with a script

Open
#2,534 1 comment 0 reactions 1 assignee Claimed by @tmeschter View on GitHub
microsoft-foundry skills untriaged
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Summary

Copilot has identified a portion of a skill that is a good candidate for replacement with a script.

The candidate is the **fine-tuned model deploy-and-verify loop** in the `microsoft-foundry` skill (`finetuning/references/deployment.md`) — an ARM token → format-lookup → PUT → poll → delete/recreate → backoff sequence whose verification half is not yet scripted.

## Candidate description

To deploy a fine-tuned model and confirm it is actually serving, the skill runs a deterministic chain:

1. `az account get-access-token` to acquire an ARM token.
2. Select `model.format` / `sku.name` / `--model-format` from a fixed lookup keyed on the base-model family (ARM and CLI surfaces differ).
3. PUT the deployment (ARM REST) or `az cognitiveservices account deployment create`.
4. Poll `provisioningState` until terminal.
5. Poll data-plane readiness; on `DeploymentNotReady`, delete + recreate + wait (~5 min), then retry first inference with exponential backoff.

This is a strong script candidate because it is:
- **Fully deterministic and output-heavy** — the caller only needs the ready endpoint URL or a clear mapped failure reason.
- **Encodes several gotchas** — ARM "Succeeded" ≠ ready, wait-after-delete, capacity ≥ 100, token refresh — currently scattered across `deployment.md` and `platform-gotchas.md`.
- **Only partly scripted today** — the existing `deploy_model.py` does the ARM PUT but **not** the readiness polling / delete-recreate / backoff verification, which is the valuable deterministic part. The repo convention also calls for **both** bash and PowerShell versions; the existing helper is Python-only.

**Sketch — `deploy-finetuned-and-verify.{sh,ps1}`:**
- **Input:** `--account`, `--resource-group`, `--deployment-name`, `--base-model`, `--capacity`.
- **Output:** the ready endpoint URL on success, or a failure reason mapped from the documented error table.

> Choosing **which** model/capacity/region to deploy, and deciding when a region is simply unsupported (`ProvisioningState: Failed`), stay in prose. Everything mechanical — token acquisition, format lookup, PUT, polling, delete→recreate→wait retry, backoff — is scriptable.

**Note — cross-area overlap:** This is the same deploy+poll archetype as the deploy-model deploy-and-poll candidate and the resource deployment monitor; the three may share one deploy-and-wait helper family.

## Affected file and lines

- [`finetuning/references/deployment.md` — CLI + ARM deploy, format table, capacity/wait rules, error table (L25–L89)](https://github.com/microsoft/GitHub-Copilot-for-Azure/blob/3890cbfb65c548ce8daa96cabd1d8de63f7bbcca/plugin/skills/microsoft-foundry/finetuning/references/deployment.md#L25-L89)
- [`finetuning/references/platform-gotchas.md` — readiness / wait-after-delete / capacity gotchas (L9, L13, L19)](https://github.com/microsoft/GitHub-Copilot-for-Azure/blob/3890cbfb65c548ce8daa96cabd1d8de63f7bbcca/plugin/skills/microsoft-foundry/finetuning/references/platform-gotchas.md#L9-L19)

## Next steps

1. **Evaluate the candidate** — confirm the steps are stable and parameterizable, and that the script captures everything the skill needs.
2. **Create both a bash _and_ a PowerShell version** of the script so the skill works across platforms.
3. **Run integration tests** to verify the scripts behave correctly and the skill still completes end-to-end.

## Background Information

### Why replace regular steps with scripts

Replacing a regular, well-defined series of steps with a script can:

- **Reduce token usage** — the skill no longer needs to spell out each command and parse large command output inline; the agent invokes one script and reads a compact result.
- **Improve reliability** — the logic is written and tested once, instead of being re-derived by the agent on every run.
- **Improve determinism** — the same inputs always produce the same steps and output, removing run-to-run variation.
- **Improve speed of execution** — a single script call replaces multiple round-trips of command generation, execution, and large-output parsing.

### Authoring notes for the scripts

- **Reference scripts with markdown links**, not just a bare path to the script file.
- **Include examples** in the skill showing how to run each script (sample invocation with arguments).
- **Briefly explain what each script does** where it is referenced.
- **The script output should explain what it did**, so the agent and user can understand the result without re-inspecting raw command output.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.