[Ticket]: 6 — Optimization & Comparative Evaluation (Days 11–12)

### Type
Performance

### Summary
We need to optimize and comparatively evaluate the mitigations from Ticket 5 to determine which provides the **best trade-off between hallucination reduction and model/system quality**, because shipping a mitigation that destroys accuracy or latency is not an improvement. Expected result: a comparative evaluation (Baseline vs Decoding vs Prompt vs Evidence/RAG vs Verification vs Abstention vs Combined) with the best-performing configuration and a documented performance/quality trade-off analysis.

### Intent
We must answer "which solution actually works best?" with one comparable table on the same evaluation set — not per-prototype numbers measured differently.

### Source of truth (links)
- Epic #1062 — Deep Analysis & Mitigation of Qwen3.5-2B Hallucinations (this ticket is its Ticket 6).
- Requester-provided epic specification (2026-08-31), Ticket 6 — "Optimization & Comparative Evaluation".
- Ticket 3 baseline metrics — the reference column.
- Ticket 5 prototypes — the candidates to compare.

### Current Behavior
- Prototypes exist but have not been compared against each other on an identical evaluation set with identical metrics.

### Expected Behavior
- A single **comparative evaluation** running all configurations on the **same evaluation set** (fixed seed):
`Baseline vs Decoding vs Prompt vs Evidence/RAG vs Verification vs Abstention vs Combined`.
- The comparative table populated:

| Configuration | Accuracy | Hallucination | Fabrication | Abstention | Latency |
|---------------|:--------:|:-------------:|:-----------:|:----------:|:-------:|
| Baseline | — | — | — | — | — |
| Decoding | — | — | — | — | — |
| Prompt | — | — | — | — | — |
| Evidence | — | — | — | — | — |
| Verification | — | — | — | — | — |
| Abstention | — | — | — | — | — |
| Combined | — | — | — | — | — |
- **Hallucination reduction** calculated for each configuration:

$$ Hallucination\ Reduction = \frac{HR_{baseline}-HR_{optimized}}{HR_{baseline}} \times 100 $$
- **Best-performing configuration** selected with justification.
- **Performance/quality trade-off analysis** (accuracy, latency, throughput, abstention) documented.

### Acceptance Criteria
- [ ] Given the prototype set from Ticket 5, when compared, then all configurations run on the identical evaluation set with identical metrics and seed.
- [ ] The full comparative table is populated for every configuration (no empty cells).
- [ ] Hallucination reduction is computed for each configuration with its formula.
- [ ] A best-performing configuration is selected and justified (not only by HR — accuracy/latency/abstention included).
- [ ] Trade-off analysis covers accuracy, latency, throughput, and abstention.
- [ ] Results are reproducible (configs + seed recorded).
- [ ] Verification evidence is provided (full results, selection rationale).

### Out of Scope
- Final validation on unseen data (Ticket 7).
- Production rollout.
- New mitigations not prototyped in Ticket 5.

### Technical Context
- Candidates: the 5 strategy families + combined prototype from Ticket 5.
- Metrics reuse the Ticket 3 definitions; latency measured per inference (TTFT/TPOT where available via gateway metrics).
- Baseline = the frozen configuration from `impl.md`.
- Live-serving caution: run at low-traffic times; `minBackends: 1`, single GPU.

### Risks
- Selection based on HR alone → quality regression. Mitigation: multi-metric selection with explicit weights/trade-off.
- Overfitting the evaluation set. Mitigation: hold out a subset for Ticket 7.
- Latency of verification/abstention loops hidden. Mitigation: measure end-to-end latency per configuration.

### Test Plan
- Execute all configurations on the shared evaluation set (fixed seed, N runs).
- Compute all metrics + hallucination reduction per configuration.
- Re-run the best configuration once to confirm stability.

### Verification evidence
- Populated comparative table (all 7 configurations × 6 columns).
- Hallucination-reduction percentages per configuration.
- Best-configuration selection rationale (trade-off analysis).
- Reproducibility details (configs, seed, dataset version).

### Human accountable owner
@stephane-segning

### AI Usage Declaration
Drafting the ticket, Proposing implementation

### Human verification completed
- [ ] I understood the intent
- [ ] I checked the source of truth
- [ ] I reviewed all AI-generated text/code
- [ ] I verified the implementation manually
- [ ] I verified the tests
- [ ] I checked for hallucinated assumptions
- [ ] I documented remaining risks
- [ ] I am the accountable owner and accept responsibility for this ticket.