Automattic / Automattic/data-liberation-agent

Detection discards its own confidence, and conflicting signals resolve by table order

Open
#130 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
31
Forks
3
Avg merge
10h 14m
Merged PRs (30d)
81

Description

`detect()` computes a confidence and a signal list, then the pipeline throws both away. Separately, when two platforms both match, the winner is whichever entry sits later in the table rather than whichever has more evidence.

## Confidence is computed and discarded

`src/lib/detect-platform/detect-platform.ts` returns `{ platform, confidence, signals }`, with `confidence` set to `high` for a header or URL match and `medium` for a source marker.

`src/lib/capture.ts` reads `.platform` and nothing else. No caller branches on confidence, nothing warns on a weak identification, and neither the confidence nor the signals reach `capture-receipt.json` or the run output. A `medium`-confidence misdetection routes the whole capture through the wrong adapter and leaves no trace of why.

## Signals do not compose

Inside `detectFromHttp`, the header loop assigns `platform` on every match rather than breaking or scoring:

```ts
for ( const sig of HTTP_SIGNALS ) {
const headerVal = response.headers.get( sig.header );
if ( headerVal && ( ! sig.value || headerVal.toLowerCase().includes( sig.value ) ) ) {
platform = sig.platform; // last match wins
confidence = 'high';
signals.push( sig.signal );
}
}
```

The source-marker loop has the same shape. Consequences:

- A page carrying markers for two platforms — common on migrated or proxied sites, and on stores embedded in another builder — resolves to whichever regex is later in the array.
- `signals` accumulates evidence from *every* platform that matched, so the returned signal list can justify a platform other than the one returned.
- Two independent weak signals for platform A cannot outweigh one signal for platform B.

The tiering itself is sound — URL, then headers, then source, then active probes, with probes correctly documented as a fallback that costs a round trip. The defect is in how matches within a tier are resolved.

## Proposed change

1. Score matches instead of overwriting: accumulate per-platform evidence within a tier, and return the highest-scoring platform with the signals that actually support it.
2. Derive `confidence` from the resulting score and margin, rather than from which tier fired.
3. Record `platform`, `confidence`, and `signals` in `capture-receipt.json` so a misdetection is diagnosable after the fact.
4. Surface a warning on the liberate path when confidence is below `high`, naming the runner-up.

## Acceptance

- A fixture serving markers for two platforms resolves deterministically to the one with more evidence, and its `signals` contain only that platform's evidence.
- The receipt carries the detection result, and `compare` / bug reports can cite it.
- A low-confidence run says so on stderr.

---

*AI assistance: researched and written by Claude via Claude Code, from reading `detect-platform.ts` and tracing its single call site. Chris Huber orchestrated and reviewed the work and is responsible for what is filed here.*

Contributor guide

Open the contributing guide

Research direction

Start in src/lib/detect-platform/detect-platform.ts, then trace its call site in src/lib/capture.ts and the liberate path. Review how capture-receipt.json is produced and how compare or bug reports consume it. Done means deterministic per-platform scoring with matching signals, persisted detection details, and a low-confidence stderr warning covered by a two-platform fixture.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.