[AI Triage] Spike: Leverage an external service for label prediction confidence scoring
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 143
Description
The goal for this spike is to complete within the agreed upon timebox and to check difficulty and feasibility of building on the work that Krista did, using an external service or another AI model for label prediction confidence scoring. We should consider:
- Whether one of the Azure service offerings is suitable for our needs
- Whether the OpenAI offering is the best fit for our needs
- Whether we should avoid using an evaluator service and instead use another AI model with dedicated prompt to score the prediction.
At the end of the spike, we should:
- Have a clear answer on "does this improve our prediction results?" and "should we consider this as a feature?"
- Understand a high-level effort for what it would take to implement this as a feature.
- If this is a potential feature, have notes or other assets that capture the approach the spike used.
- If this is a potential feature, have the reference implementation available for future inspiration.
Contributor guide
Assessment
This issue has not been assessed yet.