microsoft / microsoft/hve-core
feat(agents): add DS Eval Coach for interactive evaluation coaching and brainstorming
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 301
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 92
Description
## Summary
Proposing **DS Eval Coach**, an interactive coaching agent that helps PMs, engineers, and data scientists understand evaluation techniques, brainstorm what to measure, and explore trade-offs before committing to a formal evaluation plan.
Critically, the coach is **domain and use-case driven**: it first researches the user's specific problem space (RAG pipeline, classification task, ranking system, agent workflow, etc.) and then coaches on exactly which evaluation metrics and techniques apply to that scenario — mapping each metric to the use case it serves and explaining why it fits.
## The Problem This Solves
Evaluation is one of those areas where even experienced practitioners have doubts.
* A **PM** hears "Groundedness" and "Coherence" in a review and needs to understand what they actually measure before approving a plan.
* A **data scientist** is exploring a new domain, such as RAG or time series forecasting, and wants to brainstorm which evaluation techniques apply without reading 10 papers first.
* A **tech lead** needs to quickly compare trade-offs between metrics, such as NDCG vs. Precision@K, to make an informed decision in a design review.
* An **engineer** wants to understand why accuracy is misleading for imbalanced classes, explained with intuition and not only formulas.
The common thread is simple: teams need a knowledgeable coach to think through evaluation, not only a document generator. Sometimes they need to explore and align first, then formalize.
**Why domain research matters here:** The coach does not offer a generic list of metrics. It **researches the user's specific domain and use case first**, then coaches on which evaluation approach fits and why. For example, a RAG pipeline needs Groundedness and Context Relevance — not BLEU. A fraud detection system needs Precision-Recall tradeoff analysis — not accuracy. The agent maps each metric to the use case it serves, so teams know exactly what to measure and what each measurement tells them about their system's real-world behavior.
## How This Differs from [`eval-methodology-planner` (#1543)](https://github.com/microsoft/hve-core/issues/1543)
| Dimension | eval-methodology-planner (#1543) | Eval Coach (this proposal) |
|---|---|---|
| Purpose | Produce a versioned methodology document with provenance and readiness gates | Teach, brainstorm, and explore evaluation approaches interactively |
| When to use | Ready to commit to a formal evaluation plan | Have questions, are exploring options, or need to understand trade-offs |
| Mode | Structured planning workflow | Interactive coaching and Q&A |
| Audience | Primarily DS practitioners producing formal plans | Anyone: PMs, DS, engineers, and leads, with depth adapted to expertise |
These are complementary. A data scientist might use the coach to brainstorm approaches for an unfamiliar domain, then use the methodology planner to formalize decisions. A PM might use the coach to better understand a plan that already exists.
## Coaching Flow
```mermaid
graph LR
U[User Question] --> C[DS Eval Coach]
C --> S[Understand Context and Expertise]
S --> B[Brainstorm Metrics and Trade-offs]
B --> R[Use existing HVE Core task-researcher for current best practices]
R --> E[Explain Metrics and Interpret Scores]
E --> N[Next Step: keep exploring or hand off to eval-methodology-planner]
```
## What It Includes
| Artifact | Description |
|---|---|
| `eval-coach.agent.md` | Orchestrator that understands context and adapts depth to expertise level |
| Existing HVE Core `task-researcher` agent | Reused for live research on current evaluation best practices |
| `metric-explainer.agent.md` | Sub-agent that explains metrics at the user's level |
| `eval-knowledge` skill | 11 reference files covering classification, regression, ranking, RAG, LLM/GenAI, CV, time series, agents, fuzzy matching, loss functions, and Azure AI Foundry |
## Key Capabilities
* **Expertise-adaptive**: The same metric can be explained with analogies for a PM or with formal properties for a senior DS.
* **Brainstorming partner**: Helps explore "what should we measure" and "what are we missing" before committing to a plan.
* **Live research via existing HVE Core researcher**: Uses the current HVE Core research agent rather than introducing a duplicate research agent.
* **Metric comparison**: Supports questions like "why NDCG over MAP for our ranking use case?"
* **Interpretation help**: Supports questions like "what does a 0.7 F1 score mean for users?"
* **Domain exploration**: Helps teams entering unfamiliar areas, such as RAG, agents, and time series, understand the evaluation landscape quickly.
* **Covers 11 domains**: Classification, regression, ranking, RAG, LLM/GenAI, CV, time series, agents, fuzzy matching, loss functions, and Azure AI Foundry. More can be extended.
## Example Interactions
> "I'm a PM on a chatbot project. The DS team says we need Groundedness and Coherence. What do these mean and should I be asking about anything else?"
> "I'm building my first RAG pipeline. What evaluation techniques exist and what are the trade-offs? Help me think through this before I write a formal plan."
> "We're debating between ROUGE and BERTScore for our summarization model. Walk me through the pros and cons for our specific use case."
> "Help me understand why accuracy alone is misleading for fraud detection. I need to explain this to leadership."
## Relationship to Other Agents
| Agent | Relationship |
|---|---|
| [`eval-methodology-planner` (#1543)](https://github.com/microsoft/hve-core/issues/1543) | Complementary. Coach helps users brainstorm and understand; planner produces formal methodology documents. |
| Existing HVE Core `task-researcher` | Reused for research and evidence gathering, avoiding a duplicate research/discovery agent. |
| `eval-dataset-creator` | Upstream and downstream touchpoint. Coach helps decide what to measure; dataset creator constructs test data when needed. |
## Target Directories
* `.github/agents/data-science/eval-coach.agent.md`
* `.github/agents/data-science/metric-explainer.agent.md`
* `.github/skills/data-science/eval-knowledge/`
## Reference Implementation
Working prototype:
[README.md](https://github.com/user-attachments/files/28713496/README.md)
Happy to adapt based on maintainer feedback before submitting a PR.
Developer: Deeptanil Saha & Kartheek Palepu
Contributor guide
Assessment
This issue has not been assessed yet.