Azure / Azure/azure-sdk-tools

[Eval framework] Record a demo video walking through the Vally eval framework

Open
#16,417 0 comments 0 reactions 0 assignees View on GitHub
needs-triage
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
144

Description

Parent epic: #16344

## Goal

Produce a short demo video walking through the Vally eval framework end-to-end, for stakeholders (Praveen, Justin, Laurent, the Shanga team) who aren't in the day-to-day pipeline work.

## Suggested scope

- Discover -> shard -> run -> summarize walkthrough, both a local `vally eval` run and a CI run.
- One skill's trigger eval + a capability eval, showing a passing and a failing example.
- The specs-repo pilot (#16347) as a concrete cross-repo example.
- Where to find results today (`results.jsonl` / `eval-results.md`) and where the reporting workstream (#16416) is headed.

## Acceptance criteria

- [ ] Video recorded and hosted somewhere durable (e.g. a Stream/SharePoint link or linked from repo docs).
- [ ] Linked from the coverage/gap documentation (#16403) or the repo-root `evals/README.md` for future onboarding.
- [ ] Reviewed by at least one stakeholder listed above before being treated as final.

## Related

Part of #16344.

Contributor guide

Open the contributing guide

Research direction

Start with the Vally eval framework and run the suggested local `vally eval` flow, then inspect the CI flow and the current `results.jsonl` and `eval-results.md` outputs. Cover discover, shard, run, and summarize, including the specs-repo pilot, one trigger eval, one capability eval, and passing and failing examples. Done means a durable, stakeholder-reviewed video is linked from #16403 or the repo-root `evals/README.md`.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.