microsoft / microsoft/foundry-dev-tools
Use eval results with an llm to automatically improve an agent's system prompt
Open
@anchenyi is already working on this.
Since Jan 23, 2026.
feature request
- Dominant language
- JavaScript
- Stars
- 2.1k
- Forks
- 260
- Avg merge
- 42m
- Merged PRs (30d)
- 29
Description
Help devs use eval results to improve their agents used to create the evals automatically by providing the eval results as context to a new feature which uses an LLM to iteratively update a system prompt and re-eval.
Why this matters
- It's not clear what to do with evals right now
- Evals should help make improvements as part of the dev journey
Key scenarios this enables
- Detecting prompt regressions in agents
- Bulk model and prompt experimentation
MVP
- New tool available under the Agent and Workflow tools section, Agent Optimizer
- Select an existing eval result
- Select an agent, or provide the system prompt
- Provide a system prompt for the LLM judge or use the default one provided
- Specify the output schema for the LLM judge
- Specify the max iterations and target, runs until either one is reached
- Agent can be modified / saved directly, or system prompt updates can be copied and pasted to wherever dev has them.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.