microsoft / microsoft/GitHub-Copilot-for-Azure

Replace microsoft-foundry skill eval-dataset upload-and-register with a script

Open
#2,533 1 comment 0 reactions 1 assignee Claimed by @tmeschter View on GitHub
microsoft-foundry skills untriaged
Dominant language
Python
Stars
250
Forks
204
Avg merge
1d 12h
Merged PRs (30d)
67

Description

## Summary

Copilot has identified a portion of a skill that is a good candidate for replacement with a script.

The candidate is the **eval-dataset upload-and-register** sequence in the `microsoft-foundry` skill (`foundry-agent/eval-datasets/`) — a discover-connection → upload-blob → register → verify flow that appears near-verbatim in two files.

## Candidate description

To register a local JSONL dataset in Foundry, the skill runs a fixed operational sequence:

1. Discover the project's `AzureStorageAccount` connection and ensure the `eval-datasets` container exists.
2. `az storage blob upload` the JSONL to a deterministic `/--v.jsonl` blob path (always `--auth-mode login`).
3. Register with `evaluation_dataset_create`.
4. Verify with `evaluation_dataset_get` and display the registered name/version/URI.

This is a strong script candidate because it is:
- **A fixed CLI sequence with only parameter substitution** — account, container, blob name, file, auth mode.
- **Duplicated near-verbatim** across `trace-to-dataset.md` (Step 5) and `generate-seed-dataset.md` (Step 3); `dataset-versioning.md` points back to trace-to-dataset Step 5 as the canonical sync path.
- **Convention-enforcing** — the script can enforce the "filename starts with the agent name", "create container first if missing", and "always `--auth-mode login`" rules currently expressed as prose warnings.

**Sketch — `upload-and-register-dataset.{sh,ps1}`:**
- **Input:** `--account`, `--container` (default `eval-datasets`), `--file`, `--blob-name`, `--auth` (key|aad).
- **Output:** the ensure-container result, the upload confirmation, and the verify read-back (name/version/URI).

> The `project_connection_list` / `project_connection_create` discovery and `evaluation_dataset_create` registration are MCP tool calls that remain agent-orchestrated; choosing the storage account when no connection exists requires a user decision and stays in prose. The genuinely scriptable core is **ensure-container + blob upload + verify read-back**.

## Affected file and lines

- [`foundry-agent/eval-datasets/references/trace-to-dataset.md` — Step 5 sync (upload/register/verify) (L325–L395)](https://github.com/microsoft/GitHub-Copilot-for-Azure/blob/3890cbfb65c548ce8daa96cabd1d8de63f7bbcca/plugin/skills/microsoft-foundry/foundry-agent/eval-datasets/references/trace-to-dataset.md#L325-L395)
- [`foundry-agent/eval-datasets/references/generate-seed-dataset.md` — Step 3 register (upload variants) (L87–L134)](https://github.com/microsoft/GitHub-Copilot-for-Azure/blob/3890cbfb65c548ce8daa96cabd1d8de63f7bbcca/plugin/skills/microsoft-foundry/foundry-agent/eval-datasets/references/generate-seed-dataset.md#L87-L134)

## Next steps

1. **Evaluate the candidate** — confirm the steps are stable and parameterizable, and that the script captures everything the skill needs.
2. **Create both a bash _and_ a PowerShell version** of the script so the skill works across platforms.
3. **Run integration tests** to verify the scripts behave correctly and the skill still completes end-to-end.

## Background Information

### Why replace regular steps with scripts

Replacing a regular, well-defined series of steps with a script can:

- **Reduce token usage** — the skill no longer needs to spell out each command and parse large command output inline; the agent invokes one script and reads a compact result.
- **Improve reliability** — the logic is written and tested once, instead of being re-derived by the agent on every run.
- **Improve determinism** — the same inputs always produce the same steps and output, removing run-to-run variation.
- **Improve speed of execution** — a single script call replaces multiple round-trips of command generation, execution, and large-output parsing.

### Authoring notes for the scripts

- **Reference scripts with markdown links**, not just a bare path to the script file.
- **Include examples** in the skill showing how to run each script (sample invocation with arguments).
- **Briefly explain what each script does** where it is referenced.
- **The script output should explain what it did**, so the agent and user can understand the result without re-inspecting raw command output.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.