Azure / Azure/azure-sdk-tools

[AI Triage] Spike: Leverage an external service for label prediction confidence scoring

Open
#10,311 0 comments 0 reactions 1 assignee Claimed by @xiangyan99 View on GitHub
AI Triage Tool
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
143

Description

The goal for this spike is to complete within the agreed upon timebox and to check difficulty and feasibility of building on the work that Krista did, using an external service or another AI model for label prediction confidence scoring. We should consider:

- Whether one of the Azure service offerings is suitable for our needs
- Whether the OpenAI offering is the best fit for our needs
- Whether we should avoid using an evaluator service and instead use another AI model with dedicated prompt to score the prediction.

At the end of the spike, we should:

- Have a clear answer on "does this improve our prediction results?" and "should we consider this as a feature?"
- Understand a high-level effort for what it would take to implement this as a feature.
- If this is a potential feature, have notes or other assets that capture the approach the spike used.
- If this is a potential feature, have the reference implementation available for future inspiration.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.