investigate online evals for APIView copilot
Open
APIView Copilot
Evals
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 144
Description
- Investigate development of online evals for APIView Copilot. These should "grade" production AI comments where we don't have human eval and defer to human eval, if available.
- We can use the get comments script to retrieve comments: https://github.com/Azure/azure-sdk-tools/blob/main/packages/python-packages/apiview-copilot/README.md#getting-comments-from-apiview
- Build into existing pipeline to regularly sample prod comments for eval
Contributor guide
Assessment
This issue has not been assessed yet.