dcramer / dcramer/peated

Resolve crawler executions through an immutable run plan

Open
#1,198 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
103
Forks
18
Avg merge
1h 19m
Merged PRs (30d)
555

Description

Problem

A configured run currently resolves itself by cloning the production scraper registry and inserting a run-specific source and target. The registry is both startup composition and per-run execution state, which makes the runtime harder to reason about and tests more coupled to registry construction.

Relevant code:

  • apps/server/src/scraper/configured/runtime.ts
  • apps/server/src/scraper/runs.ts
  • apps/server/src/scraper/registry.ts
  • apps/server/src/scraper/types.ts

Desired outcome

Resolve each run once into a small immutable execution plan containing its source, targets, pinned revision behavior, and purpose. Core execution should consume that plan without mutating or cloning global composition.

Acceptance criteria

  • Define the minimum run-plan type needed by executeScraperRun.
  • Resolve code-owned and configured runs through the same narrow boundary.
  • Preserve revision pinning, target ownership, source schemas, and sink behavior.
  • Keep the production registry responsible only for code-owned definitions/startup composition.
  • Unit-test plan resolution and retain database-backed run execution tests.

Non-goals

  • Add new job types, a dependency-injection framework, or a general workflow engine.
  • Change persisted run identity or queue payloads.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading apps/server/src/scraper/configured/runtime.ts, apps/server/src/scraper/runs.ts, apps/server/src/scraper/registry.ts, and apps/server/src/scraper/types.ts, then locate executeScraperRun and the existing database-backed run execution tests. Define and resolve the minimum immutable plan for both code-owned and configured runs, preserving the listed behaviors. Done means plan-resolution unit tests pass and existing execution tests remain valid.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, testing-qa
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.