Tool Evaluations: Update dashboards to report evaluation experiment results
- Dominant language
- C#
- Stars
- 3.7k
- Forks
- 624
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 220
Description
## Summary
Update the `microsoft/github-copilot-for-azure` repository to support reporting skill evaluation experiment results to the existing dashboards, enabling visibility into tool selection quality across variants.
**Depends on:** #3035, #3036
## Requirements
### Dashboard Integration
- Ingest the structured JSON experiment results produced by the multi-variant experiment runner (see #3035)
- Surface the following in existing dashboards:
- **Per-variant metrics**: top choice success rate, average confidence, confidence distribution for baseline, namespace, and consolidated
- **Cross-variant comparisons**: deltas between variant and baseline for each experiment run
- **Trend data**: track metrics over time from nightly experiment runs to detect regressions or improvements
- **Per-area breakdown**: drill down into individual service areas (storage, keyvault, sql, etc.)
### Reporting Format
- Accept the JSON summary format defined in #3035 as input
- Support both individual experiment result files and aggregated results from full suite runs
- Provide historical comparison so new results can be evaluated against previous baselines
### Acceptance Criteria
- [ ] Existing dashboards in `microsoft/github-copilot-for-azure` updated to display experiment results
- [ ] Per-variant metrics (baseline, namespace, consolidated) visible in dashboards
- [ ] Cross-variant comparison deltas displayed
- [ ] Historical trend data from nightly runs is tracked and visualized
- [ ] Per-service-area drill-down is available
- [ ] Dashboard updates are backwards-compatible with existing reporting
Contributor guide
Assessment
This issue has not been assessed yet.