Dynamic crawler platform follow-ups
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 103
- Forks
- 18
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 555
Description
Goal
Keep Peated's dynamic crawler safe, easy to change, and understandable. This issue is only an index and ordering guide; each linked issue owns its own scope and acceptance checks.
Working rules
- Prefer small, independent changes and one pull request per issue when practical.
- Do not add new concepts or general frameworks without a current need.
- Keep old saved-rule behavior behind a clear boundary so removing it is a local change.
- Test observable behavior. Mock remote websites and model providers; use the real database and application path for lifecycle, ownership, persistence, and route behavior.
- Preserve stored IDs, run ownership, revision history, traffic limits, and source relationships.
- Keep lasting architecture notes with the crawler rather than in completed change plans.
Safety and correctness
- #1192
- #1193
- #1194
- #1195
- #1196
Start with the private-network request boundary, then operator pause behavior, then stale-worker ownership. These changes protect data and remote sites and should land before larger cleanup.
Maintainability
- #1197
- #1198
- #1199
- #1200
- #1201
Keep #1199 limited to containing today's one-source-per-site assumption. Do not implement broader source scheduling until a real source requires it.
Tests and model checks
- #1202
- #1203
The deterministic suite and live model checks remain separate. New shared test helpers should be justified by repeated setup, not introduced as a framework.
Queue boundary
Crawler runs already use the dedicated scrapers BullMQ queue with separate concurrency from default jobs. Both workers currently share one process. Splitting them into separate services is not part of this work unless measured resource or failure isolation shows a need.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
This issue is an index rather than an implementation task; start by reviewing the linked follow-ups, especially open items #1198-#1203, and their individual acceptance checks. Use the stated ordering and crawler safety rules to choose one scoped issue, with work considered done only when that linked issue's checks pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100