Automattic / Automattic/data-liberation-agent
Detection discards its own confidence, and conflicting signals resolve by table order
- Dominant language
- TypeScript
- Stars
- 31
- Forks
- 3
- Avg merge
- 10h 14m
- Merged PRs (30d)
- 81
Description
`detect()` computes a confidence and a signal list, then the pipeline throws both away. Separately, when two platforms both match, the winner is whichever entry sits later in the table rather than whichever has more evidence.
## Confidence is computed and discarded
`src/lib/detect-platform/detect-platform.ts` returns `{ platform, confidence, signals }`, with `confidence` set to `high` for a header or URL match and `medium` for a source marker.
`src/lib/capture.ts` reads `.platform` and nothing else. No caller branches on confidence, nothing warns on a weak identification, and neither the confidence nor the signals reach `capture-receipt.json` or the run output. A `medium`-confidence misdetection routes the whole capture through the wrong adapter and leaves no trace of why.
## Signals do not compose
Inside `detectFromHttp`, the header loop assigns `platform` on every match rather than breaking or scoring:
```ts
for ( const sig of HTTP_SIGNALS ) {
const headerVal = response.headers.get( sig.header );
if ( headerVal && ( ! sig.value || headerVal.toLowerCase().includes( sig.value ) ) ) {
platform = sig.platform; // last match wins
confidence = 'high';
signals.push( sig.signal );
}
}
```
The source-marker loop has the same shape. Consequences:
- A page carrying markers for two platforms — common on migrated or proxied sites, and on stores embedded in another builder — resolves to whichever regex is later in the array.
- `signals` accumulates evidence from *every* platform that matched, so the returned signal list can justify a platform other than the one returned.
- Two independent weak signals for platform A cannot outweigh one signal for platform B.
The tiering itself is sound — URL, then headers, then source, then active probes, with probes correctly documented as a fallback that costs a round trip. The defect is in how matches within a tier are resolved.
## Proposed change
1. Score matches instead of overwriting: accumulate per-platform evidence within a tier, and return the highest-scoring platform with the signals that actually support it.
2. Derive `confidence` from the resulting score and margin, rather than from which tier fired.
3. Record `platform`, `confidence`, and `signals` in `capture-receipt.json` so a misdetection is diagnosable after the fact.
4. Surface a warning on the liberate path when confidence is below `high`, naming the runner-up.
## Acceptance
- A fixture serving markers for two platforms resolves deterministically to the one with more evidence, and its `signals` contain only that platform's evidence.
- The receipt carries the detection result, and `compare` / bug reports can cite it.
- A low-confidence run says so on stderr.
---
*AI assistance: researched and written by Claude via Claude Code, from reading `detect-platform.ts` and tracing its single call site. Chris Huber orchestrated and reviewed the work and is responsible for what is filed here.*
Contributor guide
Research direction
Start in src/lib/detect-platform/detect-platform.ts, then trace its call site in src/lib/capture.ts and the liberate path. Review how capture-receipt.json is produced and how compare or bug reports consume it. Done means deterministic per-platform scoring with matching signals, persisted detection details, and a low-confidence stderr warning covered by a two-platform fixture.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100