microsoft / microsoft/benchmark-qed
Benchmarking as part of Pull Request Workflows
Open
@gaudyb is already working on this.
Since Aug 24, 2026.
enhancement
github_actions
- Dominant language
- Python
- Stars
- 98
- Forks
- 21
- Avg merge
- 9m
- Merged PRs (30d)
- 1
Description
Ha and Gaudy explored the possibility of running benchmark evaluations as part of PR checks to detect regressions, referencing similar implementations in other projects and planning to consult with Disa and Andres for further details.25:36
- PR Check Workflow: Ha described a workflow where, upon submitting a PR for a new or updated RAG system, a benchmark evaluation would be triggered automatically, and any detected regressions would flag the PR for review.
- Reference to Existing Implementations: Gaudy mentioned that Disa has implemented a similar PR check in the SOMA project and plans to review and potentially replicate this approach once Lisa is available.
- Coordination with Team Members: Ha and Gaudy agreed to consult with Andres and Disa to gather insights and technical details on implementing PR-based benchmarking checks, ensuring alignment with team practices.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.