[AI Triage] Spike: Leverage an external service for answer prediction confidence scoring
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 144
Description
The goal for this spike is to complete within the agreed upon timebox and to check difficulty and feasibility of building on the work that Krista did, using an external service or another AI model for answer prediction confidence scoring. We should consider:
- Whether one of the Azure service offerings is suitable for our needs
- Whether the OpenAI offering is the best fit for our needs
- Whether we should avoid using an evaluator service and instead use another AI model with dedicated prompt to score the prediction.
At the end of the spike, we should:
- Have a clear answer on "does this improve our prediction results?" and "should we consider this as a feature?"
- Understand a high-level effort for what it would take to implement this as a feature.
- If this is a potential feature, have notes or other assets that capture the approach the spike used.
- If this is a potential feature, have the reference implementation available for future inspiration.
Contributor guide
Research direction
No files, tests, or entry points are named in the issue. Start by locating Krista's existing answer-prediction work and compare Azure, OpenAI, and another-model approaches; done means a timeboxed comparison answers whether scoring improves predictions, estimates feature effort, and records notes plus a reference implementation if viable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- ai, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100