Azure / Azure/azure-sdk-tools

[AI Triage] Spike: Leverage an external service for answer prediction confidence scoring

Open
#10,324 0 comments 0 reactions 0 assignees View on GitHub
AI Triage Tool
Dominant language
C#
Stars
135
Forks
260
Avg merge
3d 1h
Merged PRs (30d)
144

Description

The goal for this spike is to complete within the agreed upon timebox and to check difficulty and feasibility of building on the work that Krista did, using an external service or another AI model for answer prediction confidence scoring. We should consider:

- Whether one of the Azure service offerings is suitable for our needs
- Whether the OpenAI offering is the best fit for our needs
- Whether we should avoid using an evaluator service and instead use another AI model with dedicated prompt to score the prediction.

At the end of the spike, we should:

- Have a clear answer on "does this improve our prediction results?" and "should we consider this as a feature?"
- Understand a high-level effort for what it would take to implement this as a feature.
- If this is a potential feature, have notes or other assets that capture the approach the spike used.
- If this is a potential feature, have the reference implementation available for future inspiration.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named in the issue. Start by locating Krista's existing answer-prediction work and compare Azure, OpenAI, and another-model approaches; done means a timeboxed comparison answers whether scoring improves predictions, estimates feature effort, and records notes plus a reference implementation if viable.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
ai, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.