Contain configured source, site, target, and origin cardinality
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 103
- Forks
- 18
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 555
Description
Problem
Configured source creation currently derives a site key from the hostname and creates an external site, scrape target, origin, site-target link, and single scrape source together. The database enforces one scrape source per external site. That was a deliberate pilot constraint, but the coupling is spread through creation and run lookup.
If Peated later needs reviews and catalog data from the same publisher, several sources sharing one traffic target, or one target with multiple origins, changing the cardinality will cut across the whole feature.
Relevant code:
apps/server/src/scraper/configured/service.tsapps/server/src/scraper/configured/runs.tsapps/server/src/scraper/lifecycle.tsapps/server/src/db/schema/scrapeSources.ts
Desired outcome
Keep the current one-source-per-site behavior contained behind explicit creation and lookup functions, with source identity, publisher identity, traffic policy, and allowed origin kept as separate concepts.
Acceptance criteria
- Add a durable architecture section defining source, external site, target, and origin ownership/cardinality.
- Split the monolithic creation path into small operations for those existing concepts while retaining one transaction.
- Centralize the current one-source-per-site lookup assumption instead of repeating it in lifecycle/runtime code.
- Use stored IDs and relationships for identity after creation; hostname derivation remains only an initial default.
- Add integration tests at the service boundary rather than mocking Drizzle call chains.
Non-goals
- Remove the unique constraint or implement multi-source scheduling before a concrete second-source requirement exists.
- Add a generic multi-origin crawler or new admin choices.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with apps/server/src/scraper/configured/service.ts and trace the related lookup paths in runs.ts, lifecycle.ts, and db/schema/scrapeSources.ts. Define the ownership and cardinality boundaries, then verify that creation remains transactional, runtime lookup uses stored relationships, and service-boundary integration tests cover the retained one-source-per-site behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, database
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100