Refactor FetcherBolt
- Dominant language
- Java
- Stars
- 995
- Forks
- 292
- Avg merge
- 2d 49m
- Merged PRs (30d)
- 62
Description
FetcherBolt is over 1100 lines now, FetcherThread alone is ~430 and mixes robots, sitemaps, crawl-delay, protocol calls and status emission. #867 and #504 would add more on top.
Plan: characterization tests first, then extract the queue classes as-is (same package, package-private), then thin down FetcherThread step by step. No behaviour change — anything fishy found along the way gets its own issue (first one: #2058). No new public API, extensibility stays on the streams.
Maybe later a common fetch kernel with SimpleFetcherBolt, but that's a separate discussion.
WDYT?
Contributor guide
Research direction
Start by locating FetcherBolt and FetcherThread and reading how robots, sitemaps, crawl-delay, protocol calls, and status emission are handled. Add characterization tests before extracting the queue classes, then verify behavior remains unchanged while FetcherThread is reduced and the existing test suite passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100