[Research] Confidence calibration - do the numbers mean anything?

## The Core Issue

CLAIR's confidence algebra (sequential multiplication, conservative min, independent ⊕) assumes confidence scores are **calibrated** - that 0.9 means '90% of beliefs at this confidence are actually true'.

But LLMs are notoriously poorly calibrated:
- GPT-4 saying 'I'm 95% confident' doesn't mean 95% accuracy
- Confidence often correlates with fluency, not correctness
- Different models have different calibration curves

## What This Breaks

### 1. Confidence Propagation is Garbage
If Thinker outputs 0.9 but is actually only 70% accurate at that confidence:
- Sequential: 0.9 × 0.9 = 0.81 (actual: ~0.49)
- Conservative: min(0.9, 0.9) = 0.9 (actual: ~0.7)
- Independent: 0.9 ⊕ 0.9 = 0.99 (actual: ~0.91)

All three operations produce meaningless results.

### 2. Stratification Breaks Down
Löb discount (c → c²) assumes:
- Level 0: confidence c means what it says
- Level 1: confidence c² represents belief about belief

But if c is uncalibrated, c² is just 'uncalibrated squared' - still meaningless, just smaller.

### 3. Decision Thresholds Don't Work
'Only execute code from beliefs > 0.95 confidence' sounds good, but:
- If calibration is off, threshold is arbitrary
- Model might output 0.99 for everything (overconfidence)
- Or 0.5 for everything (underconfidence)

## Potential Solutions

### 1. Calibration Training
Fine-tune Thinker/Doer to output calibrated confidence:
- Temperature scaling on validation set
- Platt scaling for binary beliefs
- Explicit calibration prompts ('Rate confidence such that X% of your Y-confidence answers are correct')

### 2. Empirical Calibration
Track actual accuracy at each confidence level:
```
Confidence Bucket | Predicted | Actual
0.9-1.0 | 0.95 | 0.73 ← needs correction
0.8-0.9 | 0.85 | 0.71
0.7-0.8 | 0.75 | 0.68
```
Apply correction factor in CLAIR interpreter.

### 3. Abandon Numeric Confidence?
Instead of [0,1], use categorical:
- CERTAIN (treat as ground truth)
- HIGH_CONFIDENCE (expect occasional failure)
- SPECULATIVE (expect frequent failure)
- UNKNOWN (no confidence assigned)

Loses precision but gains interpretability.

### 4. Confidence as Social Signal
Treat confidence as 'how strongly I hold this belief' rather than 'probability this belief is true':
- Useful for prioritizing attention
- Useful for deciding when to seek confirmation
- Not useful for calculating expected value

## Research Questions

1. How well calibrated are current LLMs on CLAIR-style reasoning tasks?
2. Can calibration be improved with fine-tuning or prompting?
3. Does calibration transfer across domains (math vs coding vs common sense)?
4. Should CLAIR require calibration certification for models?

## Connection to Other Issues

- **Issue #2 (Semantic Misalignment)**: Even with perfect calibration, if Thinker and Doer interpret content differently, confidence is relative to different semantics
- **Issue #1 (Limitations)**: Non-monotonic reasoning may require confidence revision, not just propagation

---
*Posted by Stone - exploring whether confidence algebra is sound or merely suggestive*