dcramer / dcramer/peated

Restore the scraper public boundary and durable architecture docs

Open
#1,200 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation
Dominant language
TypeScript
Stars
103
Forks
18
Avg merge
1h 19m
Merged PRs (30d)
555

Description

Problem

The scraper design calls for one public lifecycle boundary, but routes, schemas, serializers, and tests import configured/service, configured/rules, configured/runtime, and other internals directly. Durable architecture also remains mainly in completed OpenSpec change documents rather than docs/architecture and the module README.

Relevant code:

  • apps/server/src/scraper/index.ts
  • apps/server/src/scraper/README.md
  • apps/server/src/orpc/routes/external-sites/scrape-sources/
  • apps/server/src/schemas/scrapeSources.ts
  • apps/server/src/serializers/scrapeSource.ts
  • openspec/changes/define-scraper-runtime/
  • openspec/changes/add-configured-scraper-sources/

Desired outcome

Callers depend on small supported scraper APIs, while implementation modules can move or be deleted without application-wide import changes. Durable docs describe the architecture that exists today.

Acceptance criteria

  • Define explicit runtime and management entry points with only the operations/types used by outside code.
  • Move production callers off deep imports into scraper/configured/*.
  • Add a lightweight boundary check that prevents new deep production imports; test-only imports may be explicitly allowed.
  • Promote current runtime, queue, ownership, source/target/origin, versioning, and test boundaries into docs/architecture and link from the scraper README.
  • Document that scraper jobs use the dedicated scrapers queue but currently share a process with the default worker.

Non-goals

  • Create a broad barrel that exports every helper.
  • Split the worker into another service without measured resource-isolation need.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with apps/server/src/scraper/index.ts and README.md, then trace the production imports under the listed routes, schemas, and serializers. Review the two OpenSpec change directories for the current runtime and ownership boundaries. Done means supported runtime and management entry points are used by production callers, a boundary check exists, and durable architecture documentation is linked from the scraper README.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, developer-experience, documentation
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.