dcramer / dcramer/peated

Move one-off source preparation out of the configured runtime

Open
#1,201 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
103
Forks
18
Avg merge
1h 19m
Merged PRs (30d)
555

Description

Problem

apps/server/src/scraper/configured/ contains more than twenty prepare*.ts files for moving named sources to saved rules, plus a roughly 3,000-line route test. One-off rollout topology now sits beside permanent parser, runtime, and service code.

This makes the core feature look larger than it is and makes completed migration behavior expensive to remove.

Desired outcome

Permanent configured-crawler code contains permanent behavior. Bounded rollout helpers live in an operations/migration boundary and are deleted when their rollout is complete.

Acceptance criteria

  • Inventory which prepare operations are still needed in production and record evidence before removal.
  • Move still-needed one-off operations and their route behind an explicit operations/migration module.
  • Delete completed prepare modules, route branches, and tests together.
  • Split remaining tests by shared migration contract and source-specific data, avoiding one giant test file.
  • Keep the permanent configured runtime free of imports from the operations/migration module.

Non-goals

  • Change production source ownership without inventory and verification.
  • Build a generic migration framework.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inventorying the prepare*.ts files under apps/server/src/scraper/configured/ and the roughly 3,000-line route test, recording which operations are still needed in production. Trace the configured runtime imports and route branches before separating migration work. Done means completed helpers and tests are removed, remaining operations are isolated, tests are split, and the permanent runtime has no migration imports.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.