apache / apache/stormcrawler

Refactor FetcherBolt

Open
#2,057 2 comments 0 reactions 0 assignees View on GitHub
core enhancement fetcher
Dominant language
Java
Stars
995
Forks
292
Avg merge
2d 49m
Merged PRs (30d)
62

Description

FetcherBolt is over 1100 lines now, FetcherThread alone is ~430 and mixes robots, sitemaps, crawl-delay, protocol calls and status emission. #867 and #504 would add more on top.

Plan: characterization tests first, then extract the queue classes as-is (same package, package-private), then thin down FetcherThread step by step. No behaviour change — anything fishy found along the way gets its own issue (first one: #2058). No new public API, extensibility stays on the streams.

Maybe later a common fetch kernel with SimpleFetcherBolt, but that's a separate discussion.

WDYT?

Contributor guide

Open the contributing guide

Research direction

Start by locating FetcherBolt and FetcherThread and reading how robots, sitemaps, crawl-delay, protocol calls, and status emission are handled. Add characterization tests before extracting the queue classes, then verify behavior remains unchanged while FetcherThread is reduced and the existing test suite passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.