Automattic / Automattic/data-liberation-agent

Adapters should be enhancement layers over a universal discovery pipeline, not parallel implementations

Closed
#132 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
31
Forks
3
Avg merge
10h 14m
Merged PRs (30d)
81

Description

The stated purpose of this project is to liberate **any** website. The code is organised so that the "any website" path is the least developed one in the repository.

## Evidence

Discovery implementations, in lines:

| Adapter | `discover.ts` |
|---|---|
| wix | 223 |
| shopify | 210 |
| godaddy-wm | 108 |
| webflow | 101 |
| squarespace | 95 |
| weebly | 80 |
| hostinger | 79 |
| hubspot | 79 |
| **default** | **65** |

`src/adapters/default/index.ts` describes itself as reachable only through the fallback branch of `resolveAdapter`, and its `detect` always returns `false`.

The shape is parallel implementation: every adapter writes its own discovery from scratch, and the generic one is simply the version nobody invested in. A site on a platform with no adapter — which is the majority of the web, and the case the README calls "best-effort" — gets the thinnest code in the tree.

Compare this with the capture side, which already has the right shape. `AdapterCapture` in `src/adapters/page-actions.ts` is a set of optional contributions to a universal capture: `removeSelectors`, `prepare`, `responsiveImages`. Its doc comment states the principle explicitly:

> The capture path only knows that a source may have viewport-specific variants worth recording; it must not know what any particular CDN looks like.

Capture is universal with adapters layered on top. Discovery is nine parallel programs.

## Proposed direction

Invert discovery to match capture:

1. One universal discovery pipeline owned by the core — sitemap, nav-link crawl, same-origin link walk, canonicalisation, typing, and the stratified selection currently stranded in `src/adapters/shared.ts`.
2. Adapters contribute *route sources* into that pipeline (platform APIs, admin JSON endpoints, archive walkers) plus classification hints, rather than owning the whole step.
3. The generic path becomes the baseline every site gets, and a platform adapter becomes strictly additive.

Consequences worth stating up front:

- Improvements to the universal crawler benefit every unrecognised site immediately, instead of benefiting only whichever adapter is edited.
- A platform adapter can be small again. Several existing ones are mostly re-implementing sitemap and nav walking.
- The open platform-adapter PRs on this repo become smaller contributions against a stable seam rather than new parallel programs.

## Acceptance

This is a direction issue, not a patch. It is complete when adding a platform means declaring route sources and capture hints, and deleting an adapter degrades a site to the universal path rather than to a stub.

Depends on the route-inventory type being declared first, tracked separately.

---

*AI assistance: researched and written by Claude via Claude Code, from measuring the adapter tree and reading the existing `AdapterCapture` contract. Chris Huber orchestrated and reviewed the work and is responsible for what is filed here.*

Contributor guide

Open the contributing guide

Research direction

Start by reading src/adapters/shared.ts, src/adapters/default/index.ts, and src/adapters/page-actions.ts, then trace resolveAdapter and the existing discovery implementations. Compare the universal capture contract with the adapter discovery paths. Done means platform adapters contribute route sources and hints to a universal path, while removing an adapter still leaves that path working.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design, web-dev
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.