Automattic / Automattic/data-liberation-agent

Route discovery has no type: `discover()` returns `Promise<unknown>`

Open
#131 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
31
Forks
3
Avg merge
10h 14m
Merged PRs (30d)
81

Description

Route discovery is the step that decides what a liberation will contain, and it has no type. The adapter contract declares it as:

```ts
discover( url: string, opts: Record ): Promise;
```

Both ends are opaque. `src/lib/capture.ts` recovers a shape by assertion:

```ts
const inventory = ( await adapter.discover( sourceUrl, {
outputDir,
resume: options.resume === true,
} ) ) as CaptureInventory;
```

`CaptureInventory` is a private interface declared in `capture.ts`. Each of the nine adapters separately declares its own inventory type — `WixInventory`, `ShopifyInventory`, `WeeblyInventory`, `WebflowInventory`, `SquarespaceInventory`, `DefaultInventory`, and so on. No compiler check connects any of them to the shape the core asserts. An adapter can rename a field, drop `diagnostics`, or return the wrong thing entirely and the build stays green; the failure appears at runtime as a capture with zero routes.

The same opacity applies to options. `opts: Record` means the two keys the core actually passes (`outputDir`, `resume`) are conventions, not a contract.

## What is missing beyond the type

There is no discovery API at all — only this method. As a result:

- Route sources cannot compose. Sitemap parsing (`src/lib/extraction/sitemap.ts`), nav-link extraction, and adapter-specific endpoints each live inside whichever adapter happens to use them, instead of feeding one pipeline.
- A caller cannot supply routes, exclude routes, cap depth, or ask what *would* be captured before paying for a capture.
- `stratifiedUrlSlice` and `navTargetInventoryUrls` in `src/adapters/shared.ts` are general inventory primitives with no home, reachable only by adapters that remember they exist.
- `publicUrlsOnly: true` is hardcoded at the capture call site rather than being scope policy.

## Proposed change

1. Declare a `RouteInventory` type in the core — routes with `url`, `type`, provenance, plus `siteMeta` and `diagnostics` — and type `discover` as `Promise`.
2. Declare a `DiscoverOptions` interface for the option bag.
3. Move `stratifiedUrlSlice` / `navTargetInventoryUrls` and sitemap parsing into a discovery module that the core owns, so adapters contribute route sources rather than reimplementing selection.

Step 1 alone deletes the `as CaptureInventory` cast and makes all nine adapters compiler-checked against one shape.

## Acceptance

- `discover` has no `unknown` on either side.
- Removing a required field from any adapter's inventory fails the build.
- The homepage-first / pinned-nav / round-robin selection logic has exactly one implementation.

Related: the direction question of whether adapters should be enhancement layers over a universal discovery pipeline is tracked separately.

---

*AI assistance: researched and written by Claude via Claude Code, from reading the adapter contract, `capture.ts`, and all nine adapter inventory declarations. Chris Huber orchestrated and reviewed the work and is responsible for what is filed here.*

Contributor guide

Open the contributing guide

Research direction

Start with the adapter contract and src/lib/capture.ts, then compare the nine adapter inventory declarations and the shared discovery helpers. Implement the typed discovery contract and consolidate the specified discovery pieces only as far as needed. Done means no unknown in discover, adapter inventory mismatches fail the build, and selection logic has one implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design, tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.