Azure-Samples / Azure-Samples/foundry-hosted-agentframework-demos

Improve quality_eval benchmark performance and add fail-fast guardrails

Open
#10 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
47
Forks
35
PR merge metrics
No merged PRs in 30d

Description

Running uv run quality_eval.py often takes a very long time to complete (commonly around 56 to 83 minutes) and can appear to hang indefinitely.

Current behavior increases benchmark turnaround time and makes it hard to distinguish a slow healthy run from a stalled one.

Observed behavior
Recent completed quality runs for hosted-agent framework-agent showed durations of approximately:

56m 27s
79m 25s
82m 35s
2m 05s (fast-failure outlier)
Practical runtime for full runs appears to be around 60 to 80 minutes.

Contributor guide

Open the contributing guide

Research direction

Start with quality_eval.py and run it via `uv run quality_eval.py` to trace where the 56–83 minute runtime accumulates and where a run can appear stalled. Compare the reported completed-run durations and the fast-failure case, then define guardrails that distinguish slow healthy runs from stalled ones and reduce benchmark turnaround time.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.