[Adoption] Measure unassisted first success and 30-day retention

Context / problem

Onboarding evidence must answer two harder questions than install success:

1. unassisted first success — can an external engineer understand D1 and reach a meaningful capability-drift receipt without maintainer help?
2. retention — after trying it on a real project, do they keep it?

This is an evidence gate for #758/#855, not a vanity dashboard.

Current primary proposition

D1 (#856/#857):

Capture an agent's effective capability surface, commit it, and see semantically meaningful changes before deployment.

Primary first-success path:

snapshot -> inspect -> diff -> verify

Do not make the historical compiler/runtime path a prerequisite for this issue. D2/D3 require separate evidence.

What counts as first success

Not pip install, --help, a synthetic demo alone, or a maintainer-led walkthrough.

For D1, meaningful first success means the evaluator produces and understands a useful receipt from a maintained or real source, for example:

• a deterministic capability snapshot;
• a semantic diff showing capability added/removed;
• an argument/schema requiredness/type/enum change surfaced as a contract review item;
• a description-only change separated from contract drift;
• a source/provenance/digest change;
• a clear statement that ordinary Git diff was already sufficient (valid negative evidence).

The evaluator should be able to state what the output means and whether it is more useful than their current workflow.

Cohorts

Track separately:

A. High-touch design partners

Problem-discovery participants from #840. Their setup time must not be mixed into unassisted statistics.

B. Unassisted evaluators

Qualified people who use published materials asynchronously and receive no synchronous maintainer setup help before first-success measurement ends.

C. Persistent integrations

Genuine external projects that retain a D1 snapshot/diff workflow beyond evaluation. Record ~7/~30-day state where practical.

Distribution funnel

Align with #855 and #844. Preserve the upstream funnel, not only people who already decided to install:

qualified exposure
  -> understood the proposition
  -> chose to evaluate / declined
  -> attempted setup
  -> reached first useful output
  -> used on a real project
  -> retained independently / removed

Record acquisition path and decline/drop-off reason when known.

Minimum scenarios

• fresh install/checkout + maintained D1 OpenAPI fixture;
• fresh install + one realistic OpenAPI or captured MCP tools/list source;
• baseline snapshot + changed candidate + semantic diff;
• one invalid/tampered snapshot verify failure;
• removal/rollback from a real evaluation where practical.

Keep the D1 path credential-free and offline after the source file is available. Live MCP capture is separate from snapshot/inspect/diff/verify and must not be required for the basic experiment.

Metrics

Comprehension / activation

• acquisition path;
• understood D1 problem/value proposition?;
• chose to evaluate? if not, why?;
• install/setup success;
• time to first meaningful receipt;
• real-source success;
• could evaluator explain the diff without maintainer interpretation?;
• maintainer interventions required;
• comparison with current alternative: Git/config/tests/manual review.

Failure taxonomy

At minimum:

• problem not relevant/severe;
• existing alternative sufficient;
• proposition unclear;
• package/install/dependency failure;
• source configuration/normalization failure;
• unsupported source;
• semantic diff missing/misleading;
• output unclear;
• setup/integration too expensive;
• useful only as one-off diagnostic;
• other.

Retention

For genuine integrations where follow-up is permitted:

• active after ~7 days;
• active after ~30 days;
• independently operated vs maintainer-assisted;
• removed/abandoned and why;
• persistent committed snapshot / CI usage vs occasional diagnostic only;
• would removal create meaningful inconvenience?;
• did the adopter return with another use case/issue/PR/discussion?

Do not treat stars, forks or downloads as retention.

Product SLOs — experiment targets, not claims

• understand the D1 proposition in <15 seconds from the eventual README front door (#847);
• maintained fixture reaches a meaningful diff quickly enough that setup is not the experiment;
• simple real-source first value in roughly <=10 minutes;
• no synchronous maintainer help for the unassisted cohort;
• removal is easy.

Report actual outcomes; do not hide misses behind averages.

Interpretation

Strong survival evidence

• qualified external engineers reach a useful D1 result unassisted;
• real projects retain snapshot/diff usage;
• retained users can name what review/manual/risk process it improved;
• support burden remains small.

Narrow signal

• diff/inspect is useful occasionally but users reject a persistent artifact;
• only one source/workflow matters.

Response: shrink further.

Product/value failure

• evaluators understand and reach first value but consistently remove it because Git/tests/manual review are cheaper or equally good.

Response: strong D1 shrink/kill evidence.

Distribution inconclusive

• insufficient qualified exposure, unclear positioning or broken onboarding.

Response: fix distribution/onboarding once per #855; do not add architecture.

Data handling

Use the privacy-safe machinery in draft #844 or its successor. Requirements:

• no secrets, local paths or proprietary schemas in public evidence;
• anonymized cohort/project IDs supported;
• assisted vs unassisted always separate;
• synthetic fixtures never count as adoption;
• retain failures and former integrations;
• version ContextWeaver commit/release and D1 snapshot schema used.

Acceptance criteria

• evidence schema includes acquisition/exposure + D1 proposition identity;
• assisted and unassisted cohorts never merged;
• maintained D1 first-success scenario measured;
• at least one real-source D1 evaluation measured;
• failure taxonomy includes existing alternative sufficient and useful only as diagnostic;
• ~7/~30-day retention state supported for genuine integrations;
• synthetic evidence excluded from adoption counts;
• results feed #758 go/narrow/kill and #855 distribution interpretation.

Related

• #758 — controlling survival decision
• #855 — distribution-quality gate
• #856 / #857 — D1 implementation
• #840 — neutral problem discovery
• #551 — genuine adopter/retention record
• #844 — evidence foundation
• #847 — D1-first README/onboarding

Start by reading the privacy-safe evidence machinery in draft #844 and the related decision points in #758 and #855. Define the evidence schema and cohort boundaries described here, then verify that it measures maintained and real-source D1 success, preserves failure and retention states, excludes synthetic evidence, and feeds the stated go/narrow/kill decisions.