[Eval framework] Record a demo video walking through the Vally eval framework
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 144
Description
Parent epic: #16344
## Goal
Produce a short demo video walking through the Vally eval framework end-to-end, for stakeholders (Praveen, Justin, Laurent, the Shanga team) who aren't in the day-to-day pipeline work.
## Suggested scope
- Discover -> shard -> run -> summarize walkthrough, both a local `vally eval` run and a CI run.
- One skill's trigger eval + a capability eval, showing a passing and a failing example.
- The specs-repo pilot (#16347) as a concrete cross-repo example.
- Where to find results today (`results.jsonl` / `eval-results.md`) and where the reporting workstream (#16416) is headed.
## Acceptance criteria
- [ ] Video recorded and hosted somewhere durable (e.g. a Stream/SharePoint link or linked from repo docs).
- [ ] Linked from the coverage/gap documentation (#16403) or the repo-root `evals/README.md` for future onboarding.
- [ ] Reviewed by at least one stakeholder listed above before being treated as final.
## Related
Part of #16344.
Contributor guide
Research direction
Start with the Vally eval framework and run the suggested local `vally eval` flow, then inspect the CI flow and the current `results.jsonl` and `eval-results.md` outputs. Cover discover, shard, run, and summarize, including the specs-repo pilot, one trigger eval, one capability eval, and passing and failing examples. Done means a durable, stakeholder-reviewed video is linked from #16403 or the repo-root `evals/README.md`.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100