BOHICA-LABS / BOHICA-LABS/vsdd-factory
process-gap(demo-recorder+adversary): attestation quality — fabricated evidence text and presence-vs-change verification gap
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- Avg merge
- 6h 43m
- Merged PRs (30d)
- 29
Description
## Two-axis attestation quality gap
Both axes were observed in the same demo-scene delivery cycle on a private project running the factory. Each is independently reportable; they are filed together because they share a root cause: the attestation evidence generator narrated expected behavior rather than executing the story and observing actual output.
---
## Axis A: Fabricated evidence text — evidence describes architecture that never existed
### Pattern
An AC covering interactive behavior was attested by evidence text describing a specific signal/call path as the mechanism (e.g., "Signal X emitted by Component Y on input Z"). The described architecture was never present in the codebase: zero grep hits across all source files, before or after the story's merge. The actual implementation used an entirely different mechanism.
No input was dispatched. No output was observed. The evidence document read as if execution occurred and the path was confirmed. The AC was marked SATISFIED.
### Discovery
The fabrication was discovered only during a human playtest session, N weeks post-merge. A mechanical grep at that point produced 0 hits for the described component and signal. The actual mechanism was identified through source reading.
### Why this is distinct from #433
Issue #433 (no dynamic-launch attestation) covers the gap where the harness exists but is blind to a specific class of defect (e.g., input-map collision, physics/render pacing). This axis is a prior gap: **no execution occurred at all**. The evidence text described plausible architecture instead of observed behavior. The harness the #433 comments prescribe would address this by requiring evidence to be generated from actual execution — but the gap here predates harness adoption and is distinct in mechanism.
### Why this is distinct from #177
Issue #177 (hollow-demo / false-confidence checker) covers tests that execute but make vacuous assertions ("no exception," existence checks, tautologies). This axis is different: the evidence text itself is fabricated — no test/harness run backed the claim. A hollow-demo checker that reads test code would not detect this because no test code was involved; the claim lived only in prose evidence.
### Metrics
1 of 8 demo-scene ACs had fabricated evidence; discovered by first human playtest post-merge.
### Framework ask
Evidence generators for interactive behavior MUST dispatch an input (physical key, UI action, or scripted event) and quote the observed output verbatim (screenshot path, log line, signal trace). Narrative-only text describing expected code paths is not attestation. Cross-ref: #433 prescribes physical-key dispatch for input-wiring stories specifically; this axis applies to any AC claiming interactive behavior.
---
## Axis B: Presence-vs-change verification — observing field existence does not satisfy a change/increase AC
### Pattern
An AC asserted "X increases over time" (a behavioral change claim). The attestation evidence showed the field displaying its initial value. The field existed; it was visible; it read a value. The demo scene was never configured to drive the underlying system that would cause X to change, so the rate was effectively 0 for the entire session. The AC was marked SATISFIED.
### The structural error
Observing that a field EXISTS and DISPLAYS A VALUE is not evidence that the field CHANGES. A change claim requires:
1. An initial reading (T₀, value V₀).
2. A stimulus that should drive the change (the system seeded/configured to produce the behavior).
3. A second reading after elapsed time or simulation steps (T₁, value V₁ where V₁ ≠ V₀).
4. A delta assertion (V₁ > V₀ for an "increases" claim, or V₁ < V₀ for "decreases," etc.).
Single-observation evidence structurally cannot satisfy a delta AC.
### Discovery
Discovered during STORY-DEF-006 intake audit. Mechanical verification: the demo scene's initialization code contained no call to configure the system driving X; rate was 0 by inspection; a single read of 0.00 cannot distinguish "field works and starts at 0" from "field works and rate is 0" from "field renders but underlying system is not running."
### Framework ask
Any AC whose verb is change/increase/decrease/accumulate/grow requires attestation with two timestamped readings and a non-zero delta. A single reading is insufficient regardless of the field's value. Evidence generators should enforce this: for delta ACs, collect (T₀, V₀), apply stimulus, collect (T₁, V₁), assert V₁ − V₀ > 0 (or the appropriate sign), and include both readings in the evidence record.
---
## Common root cause
Both axes arise from the same gap: the framework has no contract that evidence text must be generated from actual command execution or live observation. An agent can satisfy an AC by describing what the code *should* do at runtime, which may be architecturally plausible but factually wrong.
**Proposed framework ask:** any AC gate that produces prose evidence for interactive or behavioral claims must attach the raw output from which the prose was derived — command output, screenshot path, log snippet, signal trace. If no raw output exists, the evidence fails the gate.
## Cross-refs
- #433 (harness physical-key dispatch mandate — addresses one mechanism by which A can arise once a harness exists)
- #177 (hollow-demo / false-confidence checker — vacuous-assertion class; orthogonal root cause)
- #285 (self-reported verification contradicted by independent re-execution — generalizes both axes)
- #479 (measuring-instrument stories: recorder returns void while caller expects value; silence at AC-gate; related to Axis B's presence-only check)
*(Framework-pattern detail only. From a private project running the factory.)*
Contributor guide
Assessment
This issue has not been assessed yet.