Azure-Samples / Azure-Samples/foundry-hosted-agentframework-demos
Improve quality_eval benchmark performance and add fail-fast guardrails
- Dominant language
- Python
- Stars
- 47
- Forks
- 35
- PR merge metrics
- No merged PRs in 30d
Description
Running uv run quality_eval.py often takes a very long time to complete (commonly around 56 to 83 minutes) and can appear to hang indefinitely.
Current behavior increases benchmark turnaround time and makes it hard to distinguish a slow healthy run from a stalled one.
Observed behavior
Recent completed quality runs for hosted-agent framework-agent showed durations of approximately:
56m 27s
79m 25s
82m 35s
2m 05s (fast-failure outlier)
Practical runtime for full runs appears to be around 60 to 80 minutes.
Contributor guide
Research direction
Start with quality_eval.py and run it via `uv run quality_eval.py` to trace where the 56–83 minute runtime accumulates and where a run can appear stalled. Compare the reported completed-run durations and the fast-failure case, then define guardrails that distinguish slow healthy runs from stalled ones and reduce benchmark turnaround time.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100