Meta: recurring pipeline anti-patterns surfaced by the data-unavailability deep-dive (seed for comprehensive review)

## Why this issue exists

A deep dive into *why research projects stall on data-unavailability* uncovered
three concrete defects (detailed below). But each defect is an **instance of a
more general anti-pattern** that almost certainly recurs elsewhere in the
pipeline. This issue is a **seed for a future comprehensive review** — not a
fix-it-now ticket. Nothing here should be patched in isolation; the point is to
name the recurring shapes, give a general way to *find* every instance, and a
general way to *fix* the class. Concrete other-suspected instances are listed
with `file:line` so the future review has a starting map, but that list is
deliberately lightweight and not exhaustive.

The three seed findings (all confirmed with project evidence):

1. **Two dataset-resolution mechanisms, one useless.** A weak plan-time resolver
(`src/llmxive/librarian/dataset_resolver.py::resolve_datasets`) extracts
"dataset intents" by regex on capitalized tokens near the word "dataset" — it
grabs RFC-2119 keywords (`MUST`/`SHALL`), requirement IDs (`FR-001`/`US-2`),
file-format acronyms (`CIF`/`CSV`/`VCF`), and model names (`ARIMA`/`DP-GMM`),
then fuzzy-matches each to an unrelated HuggingFace dataset (FAO → a
lip-reading set; MOCA → an anime set). Its manifest
`resolved_datasets.yaml` (written at `dataset_resolver.py:370`) is read by
**nothing** downstream.
2. **The strong resolver has a too-narrow trigger.** The real one
(`src/llmxive/librarian/data_source_discovery.py::discover_data_source`,
bridged by `src/llmxive/execution/data_source.py`) does LLM distillation →
web search → *hard* verification (fresh venv install, run a recipe, require
`RECORDS>0` + field coverage). But it fires only when a `data/` deliverable
is **missing** (`execution/data_source.py:3`, `:128`) — so a fabricating
project, whose fake data file *exists*, never triggers it.
3. **The re-plan report is failure-class-blind.**
`pipeline/graph.py::_write_execution_replan_feedback` always emits the same
advice ("replace with a CPU-tractable, dependency-light alternative",
`graph.py:1164-1167`) regardless of whether the failure was GPU, gated-data,
or a code bug.

---

## Anti-pattern 1 — Dead-end verified artifact (SSoT violation)

Something is computed/verified at one stage, persisted, but never threaded
forward — so downstream re-derives it lossily (or not at all).

**(a) Confirmed instance.** `resolved_datasets.yaml` is written at
`dataset_resolver.py:370` and read by **zero** downstream consumers (the tasker
and implementer never open it). The `resolve_datasets()` *function* is reused
in-memory by `speckit/plan_cmd.py:123,169` and `speckit/_reference_repair.py`,
but the *persisted manifest* is a pure dead-end.

**(b) Other suspected instances to audit.**
- Contrast (these look healthy — confirm during the review): `verified_facts`
is consumed in `specify_cmd.py`, `plan_cmd.py`, both revisers, and
`claims/service.py`; `grounding-cache`, `convergence-cache`, `librarian-cache`
each have a dedicated reader module. The audit should *confirm* every writer
has a reader, not assume it.
- `state/reference_repairs/` and the repair-note appended to `research.md`
(`_reference_repair.py:191,224`): verify the repaired URL is what execution
actually consumes, not re-resolved from the original rotted reference.
- `state/citations/` records: confirm the verified citation is threaded into the
compiled paper rather than re-validated from the raw reference string.

**(c) General diagnostic strategy.** For every path under `state/**` and every
`*.yaml`/`*.json` manifest a stage writes, grep for a *reader*: `grep -rn
""` across `src/`. A writer with no reader (or whose only readers are
tests) is a dead-end. Better: a tiny CI check that every state artifact class
has ≥1 non-test consumer.

**(d) General fix strategy.** Thread the verified artifact forward through the
one existing ingestion channel (SSoT) instead of persisting-and-forgetting, or
delete the write if it is genuinely unused. Never re-derive downstream what an
upstream stage already verified — pass the verified value.

---

## Anti-pattern 2 — Strong mechanism, too-narrow trigger

A good recovery mechanism is gated on a condition that rarely coincides with the
failures it should catch.

**(a) Confirmed instance.** `discover_data_source` (hard-verified real data)
only runs on a **missing** `data/` deliverable — the one case a fabricating
project avoids by writing a fake file.

**(b) Other suspected instances to audit.**
- **Kaggle GPU offload** (`execution/offload.py`, `execution/stage.py:37-47`):
fires only when a run emits a CUDA/OOM signature. Per the project memory the
planner/tasker prompts long *forbade* GPU, so the trigger signature was never
produced (0 activations ever). Trigger vs. reachable-population mismatch — the
canonical shape.
- **data-contract self-heal** (`execution/data_contract.py:231`
`find_data_contract_issues`): triggers only on specific error substrings
(missing columns / KeyError / not-in-index). Audit whether real schema
mismatches surface as other messages the matcher misses.
- **reference-repair** (`_reference_repair.py:4`): fires on the *first*
unreachable reference at the `clarified` guard — audit whether references that
rot *after* that gate (or are reachable-but-wrong) are ever re-checked.

**(c) General diagnostic strategy.** For each recovery hook, write down its
trigger predicate and the *actual population* of failures it is meant to serve;
measure overlap (activation count in `state/run-log` / `execution_status` is a
proxy — a hook that never fires is suspect). Ask: "what does a project do to
*avoid* producing this trigger?"

**(d) General fix strategy.** Broaden the trigger to the failure *class* it
targets, not one narrow syntactic signature, and add a proactive path (see
anti-pattern 3) so the mechanism doesn't depend on a rare runtime signal. Prefer
positive checks ("is the real data present and loading?") over
negative-signature matching.

---

## Anti-pattern 3 — Reactive-only, never proactive

A check that runs only *after* a costly failure, when the same check could run at
plan/task time to prevent it.

**(a) Confirmed instance.** Hard data-source verification (fresh-venv install +
recipe run + `RECORDS>0`) happens only at **execution** time, after a full
(often fabricated) run — the same verification at plan time would reject an
unavailable dataset before any compute is spent.

**(b) Other suspected instances to audit.**
- `_reference_repair.py` runs at the guard reactively; a plan-time reference
sweep would surface rotted links before tasking.
- The fabrication guard (`execution/fabrication_guard.py`) is an
execution-stage gate; consider a plan/task-time lint that flags a design whose
only data path is `random.*`/hardcoded before it is ever implemented.
- Model-tier escalation + re-plan (`graph.py:441-466`, `:1586-1623`) is entirely
post-failure; a plan-time CPU-tractability estimate could pre-empt the whole
ladder.

**(c) General diagnostic strategy.** For each expensive post-failure recovery,
ask: "could this exact check run on the plan/tasks artifacts before execution?"
If the inputs to the check exist at plan time, it is a reactive-only smell.

**(d) General fix strategy.** Hoist the check to the earliest stage where its
inputs exist (plan/tasks), keep the execution-time check as a backstop, and
share ONE implementation between the two call sites (Constitution I) — never
fork a plan-time copy.

---

## Anti-pattern 4 — Generic escape hatch that loses the specific diagnosis

A catch-all handler discards the precise failure category and emits
one-size-fits-all guidance.

**(a) Confirmed instance.** `graph.py::_write_execution_replan_feedback`
(`:1117-1174`) always tells the planner to "replace ... with a CPU-tractable,
dependency-light alternative that the free CI can run" (`:1164-1167`) — even
though `execution/stage.py:37-85` already classifies failures as
**compute-environment** (GPU/CUDA/OOM) vs **data-source-unreachable**. A GPU
failure should route to the Kaggle offload, and a gated-data failure should
trigger data-source discovery — but the re-plan report collapses all of them
into "make it smaller." The classification exists; the report throws it away.

**(b) Other suspected instances to audit.**
- `_write_convergence_replan_feedback` (`graph.py:1072`) — same "adjust the
approach" template; check it preserves *which* concerns went unresolved.
- Broad `except Exception` degrade-to-generic paths, e.g.
`execution/stage.py:672` ("data-source discovery skipped: %s") and
`_reference_repair.py:163` (degrade to `None`): audit whether the specific
cause is logged/threaded or silently flattened.

**(c) General diagnostic strategy.** Grep for catch-all handlers and
deterministic report writers (`except Exception`, `_write_*_feedback`,
`_replan`); check whether an upstream classification (like
`is_compute_env_failure` / `is_data_source_failure`) exists but is *not passed
in*. A handler that takes no failure-class argument is the tell.

**(d) General fix strategy.** Thread the failure classification into the
handler and branch guidance on it (GPU → offload/scale, gated-data →
discovery/substitute, code-bug → fix-loop). Reuse the existing classifier;
never re-derive or discard it.

---

## Anti-pattern 5 — Verifying a proxy instead of the real thing

Checking reachability of a portal/homepage, or that a file *appeared*, instead of
that the *actual needed data loads with the required fields*.

**(a) Confirmed instance.** `dataset_resolver` verifies **reachability + a
sample-format sniff** (`dataset_resolver.py:128 sniff_format`, `:162-165`) — the
URL responds and parses as *some* known format. It never checks the dataset
contains the *right* records/fields the analysis needs, which is exactly how a
`FAO` intent "verifies" against a lip-reading set. Contrast the strong path,
which requires `RECORDS>0` + field coverage against the analysis's real
`required_fields`.

**(b) Other suspected instances to audit.**
- `_reference_repair.py:145` reuses the same "reachability + a sample-format
sniff" proxy to accept a replacement URL — reachable-but-wrong-data passes.
- Execution gate history: it previously checked a file *appeared*, not its
contents (fixed via `execution/fabrication_guard.py` + `stage.py:455`
`st_size > 0`). Audit remaining gates that check existence/size but not
semantic content.
- `paper_status`/compile checks: confirm they verify the PDF *rendered the
intended content*, not merely that a PDF file exists.

**(c) General diagnostic strategy.** For every verification step, ask "what is
the *real* success condition, and what cheap proxy am I checking instead?"
Reachability, existence, non-zero size, and "parses as some format" are all
proxies. The real condition is "the specific thing the next stage needs is
present and usable."

**(d) General fix strategy.** Verify the end-to-end usable outcome: load the
data and assert the required fields/records; render the artifact and assert its
intended content. Where a hard check is expensive, make the proxy a fast
pre-filter *followed by* the real check — never the real check's replacement.

---

## Suggested scope for the future comprehensive review

- [ ] Enumerate every `state/**` artifact and manifest; confirm each writer has a
non-test reader (anti-pattern 1). Fix or delete dead-ends.
- [ ] Inventory every recovery hook (offload, data-contract self-heal,
reference-repair, discovery, fix-loop) with its trigger predicate and
measured activation count; widen triggers whose population never overlaps
real failures (anti-pattern 2).
- [ ] For each expensive post-failure check, evaluate hoisting it to plan/tasks
time with a shared implementation (anti-pattern 3).
- [ ] Thread failure classification into every re-plan / feedback writer and
branch guidance on it; stop collapsing GPU/gated-data/bug into one message
(anti-pattern 4).
- [ ] Replace every proxy verification (reachability / existence / size / "parses
as some format") with an end-to-end usable-outcome check, keeping the proxy
only as a pre-filter (anti-pattern 5).
- [ ] Cross-cutting: the two dataset-resolution mechanisms should be unified into
one strong, proactively-triggered, hard-verified resolver whose verified
output is threaded forward (touches all five anti-patterns at once).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Start by inventorying state/** artifacts and manifests, then inspect the named entry points such as dataset_resolver.py, execution/data_source.py, execution/stage.py, and pipeline/graph.py. Measure readers, recovery triggers, and failure classifications against the suggested audit scope. Done requires a documented comprehensive review with confirmed instances and agreed fixes, not isolated patches.

Written by the indexing model from the issue text.