Map: Heimdall v2 — screen-content pipeline (a11y-first + measured fallback)

## Destination

Decide the v2 screen-content pipeline. **Decided by measurement (Aug-03) and LOCKED** as the build spec in #20 (mirrors v1 spec #11): AT-SPI a11y-tree extraction is the **primary** source where available (Chromium apps need env+flag; content-bearing test ≥5 real-text nodes); **RapidOCR** the fallback for the a11y-blind remainder (per-window perceptual-hash change gate, `ocr_also` merge override); **tesseract, OmniParser-v2 and Gemma-4-E2B vision excluded** from capture-time. Media watch-sessions captured via **MPRIS** (+ CDP for the YouTube id) with watched-range transcripts — **subtitles-first** (yt-dlp signed timedtext / embedded+sidecar tracks), lazy faster-whisper CPU ASR fallback. Handed off as a locked spec; the map's next destination is implementing v2.

## Notes

- Domain: Heimdall capture pipeline. Existing flow: grim window-region JPEG → tesseract subprocess → `frames.ocr_text` → FTS5 → recap snippets (`pipes/prompts.py:130`). Capture daemon, API (:3931 via config override), pipes, CLI all live.
- Skills every session should consult: `/prototype` (HITL artifacts), `/grilling`, `/research` (AFK), `/domain-modeling`.
- Hard environment: **Intel Arc Vulkan** (no CUDA), HTTP-only machine (pip may be blocked; VPN sometimes on/off), local-only — zero cloud calls. Live infra: llama-server :8080 (Gemma-4-E2B QAT, currently `--no-mmproj`), heimdall serve :3931, capture daemon alive.
- **BENCHMARK FAIRNESS (user mandate):** identical system state before every candidate benchmark — same llama-server state, GPU/VRAM free, idle load, same corpus. If the hardware a candidate needs (e.g. CUDA for OmniParser) was not available at benchmark time, that is a reported constraint with its requirements recorded — **never a reason to bias the verdict**. Do not disqualify a horse because the hardware wasn't present.
- Real corpus: the live DB at `~/.heimdall/data.db` (85+ frames as of charting, grows daily) + JPEGs on disk; known window titles are the ground truth. Current OCR is measurably broken: frames show `Gitdub`, `heindall`, `v8.1.8` (digits/letters corrupted) and structure is entirely lost.
- Do not disturb the running serve/capture during benchmarks.
- **Screenshots stay primary for ALL windows, a11y-covered included** (user-locked): grim JPEGs remain the visual record for every capture — a11y text is an *addition* (exact text + structure), never a replacement.
- **Media watch-sessions are in scope** (user): record the YouTube URL / VLC file path per watched video, searchable; store the transcription of the *watched range only*, never the whole video.
- **Chromium/Electron a11y launch requirement (verified Aug-03):** full AT-SPI trees need BOTH `ACCESSIBILITY_ENABLED=1` (env) AND `--force-renderer-accessibility` (flag) — env-var alone stays blind (3 shell nodes, wiki-confirmed + live-verified). a11y coverage of Chrome-family apps therefore hinges on launching them with env+flag; kitty stays blind regardless. VLC transcription (#32, verified): subtitle-first (embedded tracks via `-t`, never `-to`; sidecars by filename convention), faster-whisper CPU int8 ASR as the lazy fallback (~0.13× realtime); vosk dropped; cite the archlinux.org `whisper-cpp` package page, not ArchWiki (no such wiki page exists).
- **Layered capture pipeline (decided Aug-03, measured):** a11y first (exact text + structure, ~50 ms / ~30 MB); RapidOCR for the a11y-blind remainder (3.8 s / ~573 MB → needs **frame change-detection** so unchanged frames aren't re-OCR'd); tesseract / OmniParser-v2 / Gemma excluded from capture-time. Media watch-sessions via **MPRIS** (+ CDP for the YouTube id); position is polled (µs), `Seeked` on seeks, `PropertiesChanged` on play/pause bound the session. **Prerequisites:** Chromium apps launch with `ACCESSIBILITY_ENABLED=1` + `--force-renderer-accessibility`; chrome launched with `--remote-debugging-port` (already true). **Games are the one uncovered content class** (HUD pixels — optional rare recap-time probe, not wired in).
- Refer to every ticket by name, never bare id.

## Decisions so far

- [Research: local runtime viability of fallback engines](https://github.com/adityanandanx/heimdall/issues/14) — RapidOCR v3 + Gemma-4-E2B mmproj verified runnable here (CPU / Arc-Vulkan); OmniParser-v2 installable-but-slow CPU; Surya slow; network to PyPI/HF works, Arc 11.3GB VRAM free.
- [Evaluation corpus and benchmark-fairness protocol](https://github.com/adityanandanx/heimdall/issues/15) — 30 frozen frames (chrome 12 by activity type, kitty 6, code 4, a11y-blind 6 fresh, misc 2); axes Accuracy 35 / Structure 30 / Cost 20 / Coverage 15, 3-level scoring; fairness checklist (one frozen corpus, ambient state pinned + recorded, warm-up, non-runnable = constraint not verdict); iteration rule: changes apply to all candidates' next run, never retroactive.
- [Task: assemble the frozen 30-frame corpus](https://github.com/adityanandanx/heimdall/issues/28) — 30 JPEGs + manifest at `prototypes/screen-content/bench/` (local-only, gitignored); chrome 12 / kitty 6 / code 4 / misc 2 / a11y-blind 6 (video-fullscreen, video-theatre, pdf-in-chrome, game, nvim TUI, VLC); browser is the image-viewer; system state snapshot recorded at freeze.
- [Prototype: AT-SPI accessibility-tree extraction](https://github.com/adityanandanx/heimdall/issues/16) — **a11y = v2's PRIMARY source** (exact + structured, ~50ms/30MB, weighted 0.72 on the live set); env+flag mandatory for Chromium apps; kitty off the bus (always-OCR); handoff = per-window "content-bearing nodes present → use a11y, else OCR".
- [Prototype: modern OCR baselines](https://github.com/adityanandanx/heimdall/issues/19) — **RapidOCR = the OCR fallback** (weighted 0.75; every digit tesseract garbles read exact); **tesseract does not survive** (raw 0.69, tuned 0.58 — threshold poisons dark themes); rapid's 3.8s/573MB → frame change-detection is a spec item.
- [Prototype: OmniParser-v2 as a11y-blind fallback](https://github.com/adityanandanx/heimdall/issues/17) — **excluded from capture-time** (94.7s/frame, ~18.6GB peak, garbles the game/VLC frames it exists for; weighted 0.54); icon captions are its only unique value → optional rare on-demand probe.
- [Prototype: Gemma-4-E2B vision via mmproj](https://github.com/adityanandanx/heimdall/issues/18) — **excluded from capture-time** (14–22/30 repetition loops, hallucinates code, 56–98s/frame; weighted 0.38/0.34); its video/game niche is covered by the media path → games are the uncovered corner (optional rare recap-time probe). `repeat_penalty` = untested future knob.
- [Prototype: YouTube watch-session capture](https://github.com/adityanandanx/heimdall/issues/29) — **MPRIS** {title, position µs == `video.currentTime`, status} + **CDP** {video id/url, ~0.7ms}; position polled, `Seeked`/`PropertiesChanged` bound the session.
- [Prototype: VLC watch-session capture](https://github.com/adityanandanx/heimdall/issues/30) — **MPRIS only** {exact path via `xesam:url`, title, position, length, status}, zero config; HTTP interface/a11y/argv not needed.
- [Lock the v2 screen-content pipeline spec](https://github.com/adityanandanx/heimdall/issues/20) — **the locked v2 build spec** (this body): a11y-first + RapidOCR routing, `watch_sessions` + FTS schema, merged search, subtitle-first transcription, ops prerequisites. Implementation order + open tunables below.
- [Task: live (in-progress) watch-session rows in the preview](https://github.com/adityanandanx/heimdall/issues/43) — closed `7968072`; `watch_sessions.live` (additive, in-place migration), `OpenSession.snapshot()` (pure), daemon `_live_rows`/`_sync_live_rows`, live rows finalize in place; preview shows "N live · M finished" with a watching/paused badge.

## Specified in v2 (locked in #20)

The v2 pipeline is fully specified: a11y-first source routing + `ocr_also` merge, change-gate cadence, `a11y_text`/`a11y_json`/`ocr_engine` + `watch_sessions` (+ FTS) schema with startup migration, `/search` merging frames + sessions, richer recap/time-breakdown inputs, YouTube (yt-dlp) and VLC (ffmpeg) transcription with lazy ASR, and ops prerequisites (env+flag, native-messaging extension for Chromium URLs, yt-dlp venv, optional faster-whisper). Implementation order: migration + fixture tests → source routing → API → watch daemon → transcription → pipes → CLI → ops.

## Open children — v2 implementation (spec #20 → tickets)

- [x] [Implement a11y-first text extraction (v2 source routing)](https://github.com/adityanandanx/heimdall/issues/33) — closed `b146e63`
- [x] [Implement RapidOCR fallback with ocr_also merge + change gate](https://github.com/adityanandanx/heimdall/issues/34) — closed `919df11`
- [x] [Implement MPRIS watch-session capture](https://github.com/adityanandanx/heimdall/issues/35) — closed `78c2721`; follow-up live-rows shipped `7968072` (#43)
- [x] [Implement CDP resolution for Chromium watch-sessions](https://github.com/adityanandanx/heimdall/issues/36) — closed `4ff5c6e`; persistent-connection follow-up `f0378ff`
- [x] [Replace CDP resolution with a native-messaging extension (no debug port, no approval popups)](https://github.com/adityanandanx/heimdall/issues/44) — closed `2923364`; live-verified in real Chromium (session tagged with exact URL + media_id, no popups)
- [x] [Implement merged search across frames + sessions](https://github.com/adityanandanx/heimdall/issues/37) — closed `3a03db3`; follow-up live-flag fix `efa1864`; CLI None-param fix `581a55e`
- [x] [Implement YouTube watched-range transcription](https://github.com/adityanandanx/heimdall/issues/38) — closed `2fa9df8`
- [x] [Implement VLC local-file transcription (subtitle-first)](https://github.com/adityanandanx/heimdall/issues/39) — descoped 2026-08-04 (source-only: the `xesam:url` file path is already persisted; no caption extraction)
- [x] [Implement lazy ASR fallback (faster-whisper)](https://github.com/adityanandanx/heimdall/issues/40) — closed `db0d8e1`; with #39 descoped, ASR is the transcript path for local-file sessions
- [x] [Integrate pipes with the new text sources](https://github.com/adityanandanx/heimdall/issues/41) — closed `207b289`; recap uses a11y-winner text + media session lines; breakdown's YouTube/Movies come from exact watch-session spans
- [x] [Ops: launch prerequisites, installs, and status additions](https://github.com/adityanandanx/heimdall/issues/42) — closed `ccdf245`; README ops checklist (a11y launch flags, extension install, pinned yt-dlp, optional faster-whisper); status now shows extraction mode, alive players, last session, pending ASR jobs
- [x] [Ship manual capture CLI (heimdall capture)](https://github.com/adityanandanx/heimdall/issues/45) — implementation done in tree (daemon request/ack + POST /capture + CLI), 321 tests pass, live-verified; needs commit (app.py split keeps day-browser prototype wiring uncommitted)
- [x] [Fix manual-capture re-fire storm](https://github.com/adityanandanx/heimdall/issues/46) — fixed in tree: ack keeps the consumed rid so the persistent request file cannot re-fire; regression test added; live-verified (1 frame vs 58)
- [x] [Fix a11y class-hint fallback for qualified window classes](https://github.com/adityanandanx/heimdall/issues/47) — fixed in tree: `class_hints()` substring fallback (`md.Obsidian` -> obsidian/electron hints); regression test added
- [x] [a11y collection hardening: launch flags + live verification](https://github.com/adityanandanx/heimdall/issues/48) — applied: 7 .desktop overrides + environment.d on this machine; research doc + CLASS_HINTS broadenings in tree (uncommitted)

Frontier = first open, unassigned child with all blockers closed → currently **none** — #45–#48 shipped in `a05da6a`. Remaining on the v2 roadmap: propose the next ticket (candidates: ocr_also defaults, day-browser map #21 work).

## Not yet specified

- Games corner probe (optional rare recap-time Gemma pass on game-HUD pixels) — explicitly out of v2 scope; only if a later map pass wants it.
- Opt-in `ocr_also` defaults (which window-classes ship in `window_class_merge` — code/thunar/obsidian candidates) and ASR model size at runtime — deferred to implementation, tunables in config.
- Optional rare on-demand OmniParser icon-caption probe (its only unique value) — post-v2.

## Out of scope

- Cloud OCR / vision APIs — out: heimdall is local-only.
- Full per-frame video content analysis (analyzing every frame of a recording) — out; a watched video is captured via its source + watched-range transcript, not by re-analyzing frames.