[Ticket]: 4 — Root Cause Analysis (Days 7–8)

### Type
Spike / Investigation

### Summary
We need to identify **why and under which conditions** Qwen3.5-2B hallucinates by systematically varying one factor at a time, because a mitigation (Ticket 5) is only defensible if it targets the actual causes. Expected result: a Root Cause Matrix mapping each factor to its observed effect, hallucination type, and impact, plus the comparison of every condition against the Ticket 3 baseline.

### Intent
We must answer "why does the model hallucinate in our deployment context?" with controlled experiments, not speculation. The root causes found here determine which mitigations are worth building and how much reduction is realistically achievable.

### Source of truth (links)
- Epic #1062 — Deep Analysis & Mitigation of Qwen3.5-2B Hallucinations (this ticket is its Ticket 4).
- Requester-provided epic specification (2026-08-31), Ticket 4 — "Root Cause Analysis".
- Ticket 3 baseline metrics (source of truth for comparison).
- `docs/hallucinations/` (Ticket 1) — definition, taxonomy, protocol.
- `docs/hallucinations/benchmark/` (Ticket 2) — benchmark + runner (used to execute experiments).

### Current Behavior
- Only anecdotal failure modes are known; no controlled experiment isolates any causal factor.

### Expected Behavior
- **One-factor-at-a-time experiments** varying each of:
- Model (fixed — baseline only; no other model in scope)
- Prompt (wording, structure, instructions)
- Temperature
- Top-k / Top-p
- Context length
- Context quality (relevant, noisy, irrelevant, conflicting)
- Question complexity
- Missing information
- Conflicting information
- Unknown questions
- Reasoning mode (thinking on/off)
- For every experiment, the pipeline: **Condition → Inference → Hallucination rate → Comparison with baseline**.
- A **Root Cause Matrix** (Factor / Observed effect / Hallucination type / Impact) delivered as the ticket output.

### Acceptance Criteria
- [ ] Given the baseline from Ticket 3, when a single factor is varied, then all other factors stay identical to the baseline.
- [ ] Each factor in the list is covered by at least one controlled experiment.
- [ ] Every experiment reports hallucination rate and its comparison (delta) to baseline.
- [ ] The Root Cause Matrix is complete and populated with measured values, not guesses.
- [ ] Confounding is controlled (only one factor varies per run; seeded sampling).
- [ ] Statistically meaningful sample sizes (N runs per condition, variance reported).
- [ ] Error cases are handled safely (429s/timeouts recorded).
- [ ] Verification evidence is provided (experiment outputs, matrix, variance).

### Out of Scope
- Implementing mitigations (Ticket 5).
- Model changes.
- Non-benchmark ad-hoc testing without protocol.

### Technical Context
- Runner/benchmark: Ticket 2 output; baseline metrics: Ticket 3 output.
- Sampling parameters available via the OpenAI-compatible endpoint: `temperature`, `top_p`, `top_k`, and `enable_thinking` (reasoning mode); decoding strategy knobs where supported.
- Context length experiments must respect `--max-model-len 131072` (128k) and the hybrid-architecture KV constraints (LMCache MP active; `kvCacheDtype auto`).
- The model is prone to thinking loops (Qwen model card) — flag any in experiments.

### Risks
- Factor interactions confound results. Mitigation: strict one-factor-at-a-time + baseline re-checks.
- Small samples → unstable rates. Mitigation: adequate N per condition, report confidence.
- Live serving disruption. Mitigation: low-traffic windows, rate limiting.

### Test Plan
- Execute each experiment via the runner with a fixed seed and N runs per condition.
- Re-run the baseline condition at the end of the matrix to confirm drift is negligible.
- Record all deltas in the Root Cause Matrix.

### Verification evidence
- Root Cause Matrix (Factor / Observed effect / Hallucination type / Impact), populated with measured values.
- Per-experiment outputs (config, hallucination rate, delta vs baseline, variance).

### Human accountable owner
@stephane-segning

### AI Usage Declaration
Drafting the ticket

### Human verification completed
- [ ] I understood the intent
- [ ] I checked the source of truth
- [ ] I reviewed all AI-generated text/code
- [ ] I verified the implementation manually
- [ ] I verified the tests
- [ ] I checked for hallucinated assumptions
- [ ] I documented remaining risks
- [ ] I am the accountable owner and accept responsibility for this ticket.