Automattic / Automattic/data-liberation-agent
Route discovery has no type: `discover()` returns `Promise<unknown>`
- Dominant language
- TypeScript
- Stars
- 31
- Forks
- 3
- Avg merge
- 10h 14m
- Merged PRs (30d)
- 81
Description
Route discovery is the step that decides what a liberation will contain, and it has no type. The adapter contract declares it as:
```ts
discover( url: string, opts: Record ): Promise;
```
Both ends are opaque. `src/lib/capture.ts` recovers a shape by assertion:
```ts
const inventory = ( await adapter.discover( sourceUrl, {
outputDir,
resume: options.resume === true,
} ) ) as CaptureInventory;
```
`CaptureInventory` is a private interface declared in `capture.ts`. Each of the nine adapters separately declares its own inventory type — `WixInventory`, `ShopifyInventory`, `WeeblyInventory`, `WebflowInventory`, `SquarespaceInventory`, `DefaultInventory`, and so on. No compiler check connects any of them to the shape the core asserts. An adapter can rename a field, drop `diagnostics`, or return the wrong thing entirely and the build stays green; the failure appears at runtime as a capture with zero routes.
The same opacity applies to options. `opts: Record` means the two keys the core actually passes (`outputDir`, `resume`) are conventions, not a contract.
## What is missing beyond the type
There is no discovery API at all — only this method. As a result:
- Route sources cannot compose. Sitemap parsing (`src/lib/extraction/sitemap.ts`), nav-link extraction, and adapter-specific endpoints each live inside whichever adapter happens to use them, instead of feeding one pipeline.
- A caller cannot supply routes, exclude routes, cap depth, or ask what *would* be captured before paying for a capture.
- `stratifiedUrlSlice` and `navTargetInventoryUrls` in `src/adapters/shared.ts` are general inventory primitives with no home, reachable only by adapters that remember they exist.
- `publicUrlsOnly: true` is hardcoded at the capture call site rather than being scope policy.
## Proposed change
1. Declare a `RouteInventory` type in the core — routes with `url`, `type`, provenance, plus `siteMeta` and `diagnostics` — and type `discover` as `Promise`.
2. Declare a `DiscoverOptions` interface for the option bag.
3. Move `stratifiedUrlSlice` / `navTargetInventoryUrls` and sitemap parsing into a discovery module that the core owns, so adapters contribute route sources rather than reimplementing selection.
Step 1 alone deletes the `as CaptureInventory` cast and makes all nine adapters compiler-checked against one shape.
## Acceptance
- `discover` has no `unknown` on either side.
- Removing a required field from any adapter's inventory fails the build.
- The homepage-first / pinned-nav / round-robin selection logic has exactly one implementation.
Related: the direction question of whether adapters should be enhancement layers over a universal discovery pipeline is tracked separately.
---
*AI assistance: researched and written by Claude via Claude Code, from reading the adapter contract, `capture.ts`, and all nine adapter inventory declarations. Chris Huber orchestrated and reviewed the work and is responsible for what is filed here.*
Contributor guide
Research direction
Start with the adapter contract and src/lib/capture.ts, then compare the nine adapter inventory declarations and the shared discovery helpers. Implement the typed discovery contract and consolidate the specified discovery pieces only as far as needed. Done means no unknown in discover, adapter inventory mismatches fail the build, and selection logic has one implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend-api-design, tooling
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100