microsoft / microsoft/foundry-dev-tools
Create an eval from traces
Open
Nobody has claimed this yet.
feature request
- Dominant language
- JavaScript
- Stars
- 2.1k
- Forks
- 260
- Avg merge
- 42m
- Merged PRs (30d)
- 29
Description
Add new features that make simpler to create evals by using traces as the dataset. Example:
Why this matters
- Evals are important to get the most out of AI systems but are incredibly time consuming and frustrating to create
- It can be confusing on what shape the data needs to be in for different eval types
Key scenarios this enables
- Detecting prompt regressions in agents
- Bulk model and prompt experimentation
MVP
- The
InputandOutputcan be automatically mapped to eval variables{{query}}and{{response}}. This is an assumption to streamline the experience. The same may be true for mapping totool_definitionsandtool_calls. - Start with options that easily map to trace data like Relevance, Task Adherence, Coherence, Similarity, Intent Resolution, and Tool Call Accuracy.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by tracing how datasets, traces, and eval types are currently represented, then compare the MVP scenarios and mappings against those flows; done means creating an eval from trace data with the listed variable mappings and supported eval options.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100