microsoft / microsoft/foundry-dev-tools

Use eval results with an llm to automatically improve an agent's system prompt

Open
#292 2 comments 0 reactions 1 assignee View on GitHub

@anchenyi is already working on this.

Since Jan 23, 2026.

feature request
Dominant language
JavaScript
Stars
2.1k
Forks
260
Avg merge
42m
Merged PRs (30d)
29

Description

Help devs use eval results to improve their agents used to create the evals automatically by providing the eval results as context to a new feature which uses an LLM to iteratively update a system prompt and re-eval.

Image

Why this matters

  • It's not clear what to do with evals right now
  • Evals should help make improvements as part of the dev journey

Key scenarios this enables

  • Detecting prompt regressions in agents
  • Bulk model and prompt experimentation

MVP

  • New tool available under the Agent and Workflow tools section, Agent Optimizer
  • Select an existing eval result
  • Select an agent, or provide the system prompt
  • Provide a system prompt for the LLM judge or use the default one provided
  • Specify the output schema for the LLM judge
  • Specify the max iterations and target, runs until either one is reached
  • Agent can be modified / saved directly, or system prompt updates can be copied and pasted to wherever dev has them.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.