[Epic]: Deep Analysis & Mitigation of Qwen3.5-2B Hallucinations

### Executive Summary

We want to systematically characterize, measure, diagnose and mitigate the hallucinations of the self-hosted **`Qwen/Qwen3.5-2B`** fleet coding model under its **current deployment configuration** (vLLM `-cu129`, 1× RTX 4000 SFF Ada, BF16, 128k context, LMCache MP) because hallucinated code, APIs and technical claims directly degrade developer trust, output correctness and the business case of the self-hosted fleet.

This epic exists to solve: the absence of an objective, measured understanding of **what** Qwen3.5-2B hallucinates, **how often**, **under which conditions** and **why**, and the absence of an evidence-based, validated mitigation to recommend for production.

It does not exist to merely produce code, tickets, documentation, or AI-generated artifacts.

### Strategic Intent

> Measure Qwen3.5-2B hallucinations objectively on our exact deployment configuration, trace them to root causes, and ship the best-validated mitigation as a recommended production configuration — demonstrated with data, not opinion.

A person must be able to say this verbally without reading this document.

### Problem Statement

Currently:
- `qwen3-5-2b` is the fleet coding model (deployed 2026-08-10, PR #970, replacing `qwen25-3b-awq`), yet **no hallucination baseline exists** for our exact serving configuration (vLLM, BF16, 128k, LMCache MP).
- Known failure modes are **anecdotal, not measured**: long-context latency degradation (P6 prefix p50 1827 ms vs SLO < 800 ms), thinking loops, tool-call parser `qwen3_coder`, LMCache MP interplay — none evaluated against hallucination behavior.
- The optimization backlog (fp8 KV cache, 256k window, higher concurrency, MTP, text-only) has **never been scored for hallucination impact**.

This causes:
- Impact on users: developers acting on hallucinated code, library APIs, or technical claims.
- Impact on developers/operations: no evidence base for deciding which lever (decoding / prompt / grounding / verification / abstention) to pull.
- Impact on business/security/compliance: unmeasured quality of the self-hosted fleet, and no defensible metric to justify it.

### Desired Outcome

When this epic is complete:
- A **frozen, reproducible baseline**: the deployment configuration (from `impl.md`) plus the experimental protocol.
- A **versioned benchmark + automated inference runner + raw response dataset** representative of real coding usage.
- **Baseline metrics**: hallucination rate, factual accuracy, fabrication rate, unsupported-claim rate, abstention accuracy, false-confidence rate, severity distribution.
- A **root-cause matrix** (factor → observed effect → hallucination type → impact).
- **Implemented and evaluated mitigations**: decoding, prompt, evidence grounding, verification, abstention, combined.
- A **final recommendation**: one validated configuration, tested on unseen data, with documented latency/throughput impact.

Success means that we can demonstrate with data: **what hallucinates → how often → under which conditions → why → how to mitigate → how much improvement → what configuration to use**.

### Scope (In / Out)

#### In Scope
- Hallucination definition, taxonomy (categories) and severity levels for the coding/technical use case.
- Experimental protocol, reproducibility requirements, per-inference data collection.
- Versioned benchmark + automated, reproducible inference runner.
- Characterization & baseline measurement (Ticket 3).
- Root-cause analysis, one-factor-at-a-time (Ticket 4).
- Mitigation prototypes: decoding, prompt, evidence grounding, verification, abstention, combined (Ticket 5).
- Comparative optimization & performance/quality trade-off analysis (Ticket 6).
- Final validation on unseen data + production recommendation (Ticket 7).

#### Out of Scope
- Fine-tuning / retraining / changing the weights or architecture of Qwen3.5-2B.
- Changes to vLLM source code or the serving engine.
- Production rollout of the chosen mitigation (delivered as a recommendation; rollout is a follow-up).
- Hallucination analysis of any other model.
- Changes to the `charts/inference` engine profiles or chart logic.

Anything not listed in scope must be clarified before implementation.

### Source of truth (links)
- Requester-provided epic specification **"Deep Analysis & Mitigation of Qwen3.5-2B Hallucinations"** (2026-08-31): 7 tickets × 2 days, methodology `Characterize → Reproduce → Measure → Diagnose → Analyze Causes → Mitigate → Validate`.
- `impl.md` (repo root) — frozen Qwen3.5-2B deployment configuration (baseline to be committed).
- `context.md` — deployment context (model, fleet, serving stack, LMCache history, validation).
- `plans/qwen3-5-2b-replacing-qwen25-3b-awq.md` — hyperparameters, 7-step deployment plan, risks, rollback.
- #973 — Test Qwen3.5-2B, update documentation, optimize deployment.
- #1020 — KV-cache offloading for LLM inference.
- `docs/patterns/lmcache-mp-serving.md` — LMCache MP activation & working state (2026-08-25).
- ADRs: 0095 (cluster-local federation), 0097 (engine-agnostic hardening), 0101 (load gate before federation), 0104 (GPU cost basis), 0118 (fp8 KV cache, LMCache ⇒ `auto`), 0126/0129 (catalogs in values repo), 0128 (nominal pricing).

### Stakeholders

| Role | Name | Responsibility |
|------|------|---------------|
| Product Owner | @stephane-segning | Owns product intent and priority |
| Technical Lead | @stephane-segning | Owns architecture and technical direction |
| Delivery Owner | @stephane-segning | Owns planning and coordination |
| Security / Compliance | @stephane-segning | Owns risk constraints |
| Engineering Team | ai-helm + inference-ops | Owns implementation and verification |

### Key Assumptions
- The frozen baseline in `impl.md` accurately reflects the deployed configuration on `hetzner-k8s-gpu-2` (ns `inference`).
- Hallucinations are reproducible enough to measure (seeded sampling, N runs per case).
- The benchmark is representative of real coding usage of `qwen3-5-2b-local`.
- Ground truth can be defined unambiguously for the chosen test cases.
- All mitigations are testable through the existing OpenAI-compatible endpoint without infrastructure changes.

Each assumption must be validated, rejected, or converted into a risk before delivery.

### Constraints
- Technical: vLLM nightly `-cu129`, single RTX 4000 SFF Ada per model, **one model per GPU**; no vLLM source changes; benchmark runs must not disrupt live serving.
- Security: cluster-local model, API key via ExternalSecret; no exfiltration of prompts/responses; the inference dataset must not contain secrets.
- Compliance: every artifact follows the ADORSYS-GIS AI governance templates; AI output is not truth; human verification is mandatory.
- Timeline: 2 weeks — 7 tickets × 2 days (2026-09-01 → 2026-09-18).
- Dependency: GPU cluster access (`hetzner-k8s-gpu-2`), storage for the dataset, benchmark tooling.
- Operational: `minBackends: 1` — serving is single-instance; benchmark load must be scheduled to avoid disruption.

### Non-Functional Requirements
This epic must consider:
- Performance: benchmark runs must not disrupt live serving (single GPU, `minBackends: 1`); latency/throughput impact of any recommended mitigation must be measured.
- Security: cluster-local model, API key via ExternalSecret; no exfiltration of prompts/responses; inference dataset must not contain secrets.
- Reliability: reproducible, seeded experiments; results reproducible by another engineer.
- Observability: per-inference metadata (latency, token usage, seed, sampling params); metrics recorded in the dataset.
- Maintainability: benchmark, runner, and analysis documented and versioned in the repo (`docs/hallucinations/`).
- Data privacy: prompts and contexts scrubbed of secrets before storage.

### Risks

| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|------------|
| GPU access unavailable during the sprint | Medium | High | Reserve access up front; batch benchmark runs |
| Hallucinations not reproducible (high variance) | Medium | Medium | Seeded sampling, N runs per case, report variance |
| Ground-truth ambiguity in test cases | Medium | Medium | Annotation rules, dual annotation, pre-defined edge cases |
| Benchmark load degrades live serving | Low | High | Run via the gateway in low-traffic windows; monitor GPU/429s |
| Mitigation reduces quality / latency / throughput | Medium | High | Multi-metric evaluation (accuracy, latency, throughput, abstention) |
| AI-generated benchmark/analysis introduces bias | Medium | Medium | Human review per ticket; governance verification checklist |

### Success metrics

| Metric | Current | Target | Source |
|--------|---------|--------|--------|
| Hallucination rate (HR) | Not measured | Baseline established; measurable reduction after mitigation | Benchmark runner (Ticket 3/6) |
| Factual accuracy | Not measured | Baseline; no significant regression after mitigation | Benchmark runner |
| Fabrication rate | Not measured | Baseline; reduction after mitigation | Benchmark runner |
| Abstention accuracy | Not measured | Baseline; improved with abstention mitigation | Benchmark runner |
| Hallucination reduction `(HR_baseline − HR_opt) / HR_baseline × 100` | — | ≥ 30% (hypothesis to validate) | Comparative evaluation (Ticket 6) |
| Latency impact of recommended config | P6 p50 1827 ms (LMCache-off baseline) | No significant regression vs baseline | Benchmark + gateway metrics |
| Recommended configuration delivered | None | One validated config + reproduction guide | Final report (Ticket 7) |

Avoid vanity metrics. Measure outcomes, not activity.

### Child User Stories
- [ ] #1063 — Ticket 1 · Define the Hallucination Problem & Experimental Baseline (Days 1–2)
- [ ] #1064 — Ticket 2 · Build the Hallucination Benchmark & Reproduction Pipeline (Days 3–4)
- [ ] #1065 — Ticket 3 · Characterize & Measure Hallucinations (Days 5–6)
- [ ] #1066 — Ticket 4 · Root Cause Analysis (Days 7–8)
- [ ] #1067 — Ticket 5 · Design & Implement Mitigation Strategies (Days 9–10)
- [ ] #1068 — Ticket 6 · Optimization & Comparative Evaluation (Days 11–12)
- [ ] #1069 — Ticket 7 · Final Validation & Recommendation (Days 13–14)

Each child ticket must have its own acceptance criteria and verification evidence.

### Human accountable owner
@stephane-segning

### AI Usage Declaration
Drafting, Summarization, Ticket decomposition, Technical proposal

### Human verification completed
- [ ] Intent checked against source of truth
- [ ] Scope reviewed by Product Owner
- [ ] Technical feasibility reviewed by Technical Lead
- [ ] Risks reviewed
- [ ] Acceptance criteria reviewed
- [ ] No unverified AI claim remains
- [ ] I am the accountable owner and accept responsibility for this epic.

### Definition of Ready
This epic is ready only when:
- [ ] The strategic intent is clear
- [ ] Scope and out-of-scope are explicit
- [ ] Source of truth is linked
- [ ] Stakeholders are identified
- [ ] Major risks are documented
- [ ] Success metrics are defined
- [ ] Child tickets are identified or planned
- [ ] Human owner has reviewed all AI-generated content

### Definition of Done
This epic is done only when we can demonstrate with data:

> **What hallucinations occur → how often → under which conditions → why → how to mitigate them → how much improvement we obtain → what configuration should be used.**

And:
- [ ] All child tickets (#1063–#1069) are completed
- [ ] Acceptance criteria are satisfied
- [ ] Success metrics have been measured or scheduled for measurement
- [ ] Documentation is updated
- [ ] Operational impact is reviewed
- [ ] Security/compliance impact is reviewed
- [ ] Stakeholders have accepted the result
- [ ] Lessons learned are documented