azurenoops / azurenoops/spin_agent
[HIGH-5] Fabricated confidence scores
- Dominant language
- C#
- Stars
- 3
- Forks
- 1
- Avg merge
- 18h 9m
- Merged PRs (30d)
- 43
Description
## Problem
Confidence scores displayed to ISSOs (and potentially included in ATO artifacts) are hardcoded to `0.8` and `0.6`. They are presented as if they were reliability-derived metrics, but they are constants with no connection to model output probability, retrieval confidence, or any runtime signal. This is misleading to compliance officers who may use these figures when making authorization decisions.
## Evidence / Location
- Confidence score assignment in compliance response builder / narrative generator
- Values `0.8` (primary) and `0.6` (secondary/fallback) are literals, not computed
## Fix
1. Either compute confidence from real signals (model logprobs, retrieval score, citation count) or remove the field entirely.
2. If a placeholder is required short-term, label it explicitly as `"confidence": null` or `"confidence": "not-computed"` — never return a fake numeric value.
3. Document the intended confidence computation contract in the API schema.
## Source Reviewers
Vision · Banner
---
> ⚠️ **Caveat:** All findings are static read-reviews; build/tests were not run. Reproduce before treating fix as confirmed.
> 📋 **Note:** War Machine's ATO/RMF findings are still pending and will be folded in as a second batch.
Contributor guide
Assessment
This issue has not been assessed yet.