dcramer / dcramer/peated

Dynamic crawler platform follow-ups

Open
#1,210 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
103
Forks
18
Avg merge
1h 19m
Merged PRs (30d)
555

Description

Goal

Keep Peated's dynamic crawler safe, easy to change, and understandable. This issue is only an index and ordering guide; each linked issue owns its own scope and acceptance checks.

Working rules

  • Prefer small, independent changes and one pull request per issue when practical.
  • Do not add new concepts or general frameworks without a current need.
  • Keep old saved-rule behavior behind a clear boundary so removing it is a local change.
  • Test observable behavior. Mock remote websites and model providers; use the real database and application path for lifecycle, ownership, persistence, and route behavior.
  • Preserve stored IDs, run ownership, revision history, traffic limits, and source relationships.
  • Keep lasting architecture notes with the crawler rather than in completed change plans.

Safety and correctness

  • #1192
  • #1193
  • #1194
  • #1195
  • #1196

Start with the private-network request boundary, then operator pause behavior, then stale-worker ownership. These changes protect data and remote sites and should land before larger cleanup.

Maintainability

  • #1197
  • #1198
  • #1199
  • #1200
  • #1201

Keep #1199 limited to containing today's one-source-per-site assumption. Do not implement broader source scheduling until a real source requires it.

Tests and model checks

  • #1202
  • #1203

The deterministic suite and live model checks remain separate. New shared test helpers should be justified by repeated setup, not introduced as a framework.

Queue boundary

Crawler runs already use the dedicated scrapers BullMQ queue with separate concurrency from default jobs. Both workers currently share one process. Splitting them into separate services is not part of this work unless measured resource or failure isolation shows a need.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

This issue is an index rather than an implementation task; start by reviewing the linked follow-ups, especially open items #1198-#1203, and their individual acceptance checks. Use the stated ordering and crawler safety rules to choose one scoped issue, with work considered done only when that linked issue's checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.