Move one-off source preparation out of the configured runtime
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 103
- Forks
- 18
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 555
Description
Problem
apps/server/src/scraper/configured/ contains more than twenty prepare*.ts files for moving named sources to saved rules, plus a roughly 3,000-line route test. One-off rollout topology now sits beside permanent parser, runtime, and service code.
This makes the core feature look larger than it is and makes completed migration behavior expensive to remove.
Desired outcome
Permanent configured-crawler code contains permanent behavior. Bounded rollout helpers live in an operations/migration boundary and are deleted when their rollout is complete.
Acceptance criteria
- Inventory which prepare operations are still needed in production and record evidence before removal.
- Move still-needed one-off operations and their route behind an explicit operations/migration module.
- Delete completed prepare modules, route branches, and tests together.
- Split remaining tests by shared migration contract and source-specific data, avoiding one giant test file.
- Keep the permanent configured runtime free of imports from the operations/migration module.
Non-goals
- Change production source ownership without inventory and verification.
- Build a generic migration framework.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inventorying the prepare*.ts files under apps/server/src/scraper/configured/ and the roughly 3,000-line route test, recording which operations are still needed in production. Trace the configured runtime imports and route branches before separating migration work. Done means completed helpers and tests are removed, remaining operations are isolated, tests are split, and the permanent runtime has no migration imports.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100