Automattic / Automattic/data-liberation-agent
Adapters should be enhancement layers over a universal discovery pipeline, not parallel implementations
- Dominant language
- TypeScript
- Stars
- 31
- Forks
- 3
- Avg merge
- 10h 14m
- Merged PRs (30d)
- 81
Description
The stated purpose of this project is to liberate **any** website. The code is organised so that the "any website" path is the least developed one in the repository.
## Evidence
Discovery implementations, in lines:
| Adapter | `discover.ts` |
|---|---|
| wix | 223 |
| shopify | 210 |
| godaddy-wm | 108 |
| webflow | 101 |
| squarespace | 95 |
| weebly | 80 |
| hostinger | 79 |
| hubspot | 79 |
| **default** | **65** |
`src/adapters/default/index.ts` describes itself as reachable only through the fallback branch of `resolveAdapter`, and its `detect` always returns `false`.
The shape is parallel implementation: every adapter writes its own discovery from scratch, and the generic one is simply the version nobody invested in. A site on a platform with no adapter — which is the majority of the web, and the case the README calls "best-effort" — gets the thinnest code in the tree.
Compare this with the capture side, which already has the right shape. `AdapterCapture` in `src/adapters/page-actions.ts` is a set of optional contributions to a universal capture: `removeSelectors`, `prepare`, `responsiveImages`. Its doc comment states the principle explicitly:
> The capture path only knows that a source may have viewport-specific variants worth recording; it must not know what any particular CDN looks like.
Capture is universal with adapters layered on top. Discovery is nine parallel programs.
## Proposed direction
Invert discovery to match capture:
1. One universal discovery pipeline owned by the core — sitemap, nav-link crawl, same-origin link walk, canonicalisation, typing, and the stratified selection currently stranded in `src/adapters/shared.ts`.
2. Adapters contribute *route sources* into that pipeline (platform APIs, admin JSON endpoints, archive walkers) plus classification hints, rather than owning the whole step.
3. The generic path becomes the baseline every site gets, and a platform adapter becomes strictly additive.
Consequences worth stating up front:
- Improvements to the universal crawler benefit every unrecognised site immediately, instead of benefiting only whichever adapter is edited.
- A platform adapter can be small again. Several existing ones are mostly re-implementing sitemap and nav walking.
- The open platform-adapter PRs on this repo become smaller contributions against a stable seam rather than new parallel programs.
## Acceptance
This is a direction issue, not a patch. It is complete when adding a platform means declaring route sources and capture hints, and deleting an adapter degrades a site to the universal path rather than to a stub.
Depends on the route-inventory type being declared first, tracked separately.
---
*AI assistance: researched and written by Claude via Claude Code, from measuring the adapter tree and reading the existing `AdapterCapture` contract. Chris Huber orchestrated and reviewed the work and is responsible for what is filed here.*
Contributor guide
Research direction
Start by reading src/adapters/shared.ts, src/adapters/default/index.ts, and src/adapters/page-actions.ts, then trace resolveAdapter and the existing discovery implementations. Compare the universal capture contract with the adapter discovery paths. Done means platform adapters contribute route sources and hints to a universal path, while removing an adapter still leaves that path working.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend-api-design, web-dev
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100