Restore the scraper public boundary and durable architecture docs
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 103
- Forks
- 18
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 555
Description
Problem
The scraper design calls for one public lifecycle boundary, but routes, schemas, serializers, and tests import configured/service, configured/rules, configured/runtime, and other internals directly. Durable architecture also remains mainly in completed OpenSpec change documents rather than docs/architecture and the module README.
Relevant code:
apps/server/src/scraper/index.tsapps/server/src/scraper/README.mdapps/server/src/orpc/routes/external-sites/scrape-sources/apps/server/src/schemas/scrapeSources.tsapps/server/src/serializers/scrapeSource.tsopenspec/changes/define-scraper-runtime/openspec/changes/add-configured-scraper-sources/
Desired outcome
Callers depend on small supported scraper APIs, while implementation modules can move or be deleted without application-wide import changes. Durable docs describe the architecture that exists today.
Acceptance criteria
- Define explicit runtime and management entry points with only the operations/types used by outside code.
- Move production callers off deep imports into
scraper/configured/*. - Add a lightweight boundary check that prevents new deep production imports; test-only imports may be explicitly allowed.
- Promote current runtime, queue, ownership, source/target/origin, versioning, and test boundaries into
docs/architectureand link from the scraper README. - Document that scraper jobs use the dedicated
scrapersqueue but currently share a process with the default worker.
Non-goals
- Create a broad barrel that exports every helper.
- Split the worker into another service without measured resource-isolation need.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with apps/server/src/scraper/index.ts and README.md, then trace the production imports under the listed routes, schemas, and serializers. Review the two OpenSpec change directories for the current runtime and ownership boundaries. Done means supported runtime and management entry points are used by production callers, a boundary check exists, and durable architecture documentation is linked from the scraper README.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, developer-experience, documentation
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100