apache/stormcrawler

View on GitHub

A scalable, mature and versatile web crawler based on Apache Storm

Stars: 995
Forks: 292
Open beginner issues: 0
Indexed issues: 30
Avg merge: 2d 49m
Merged PRs (30d): 62

Dominant language: Java
License: Apache-2.0
Last GitHub push: Sep 13, 2026
Latest indexed: Sep 14, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: good first issue help wanted

Open All

30 issues indexed so far

URLPartitionerBolt to partition on arbitrary metadata value Open

core enhancement good first issue

apache/stormcrawler#85 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Normalise HTTP headers Open

core fetcher good first issue help wanted

apache/stormcrawler#108 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Remove needsDOM() in ParseFilter Open

good first issue help wanted question

apache/stormcrawler#158 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Adaptative URL filter to normalize URLs based on canonical tag Open

good first issue

apache/stormcrawler#315 · 6 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Capture stats about hosts on status stream post fetching Open

good first issue ideas

apache/stormcrawler#504 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
wish: WARCHdfsBolt with CDX index Open

good first issue help wanted warc wish

apache/stormcrawler#567 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
New XPATH implementation based on JSON file and organised per hostname or domain Open

core enhancement good first issue

apache/stormcrawler#586 · 0 comments · 1 reaction · 0 assignees · Indexed Sep 14, 2026
Send host info on a specific stream Closed

enhancement good first issue help wanted

apache/stormcrawler#867 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Delete redirected pages Open

core good first issue

apache/stormcrawler#994 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
[Improvement] Work towards Maven 4 Compatibility Open

good first issue help wanted

apache/stormcrawler#1662 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 14, 2026
[Improvement] Add configuration to stop parsing PDFs after X pages Open

enhancement

apache/stormcrawler#1901 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Refactor FetcherBolt Open

core enhancement fetcher

apache/stormcrawler#2057 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
fetcher.timeout.queue silently drops URLs Open

bug core fetcher

apache/stormcrawler#2058 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
http.basicauth credentials are attached to every request, whatever the host Open

apache/stormcrawler#2089 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
A response header named set-header becomes a request header directive Open

apache/stormcrawler#2090 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Response headers can populate the reserved verbatim request and response metadata keys Open

apache/stormcrawler#2091 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
CharsetIdentification retries the meta charset lookup ten bytes at a time by recursing, and the call sits outside JSoupParserBolt's catch block Open

apache/stormcrawler#2092 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
AbstractLLMTextExtractor pastes page HTML into the prompt unescaped and returns the reply unchecked Open

apache/stormcrawler#2093 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Playwright browser context stops validating certificates as soon as a proxy is configured Open

apache/stormcrawler#2094 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Playwright protocol applies neither the IP filter nor the URL filters to the requests it makes Open

apache/stormcrawler#2095 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Tika ParserBolt puts no limit on the extracted text and no time limit on the parse Open

apache/stormcrawler#2097 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
URLFrontier channels are always plaintext and the module has no TLS option Open

apache/stormcrawler#2098 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Response headers named metrics.* are fed straight into the fetcher metric registry, and a non-numeric one fails the fetch Open

apache/stormcrawler#2099 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Gzipped sitemap files are expanded with no size bound, and SiteMapParserBolt reads Content-Type under a key that is never present Open

apache/stormcrawler#2100 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
HostURLFilter memoizes the source host in plain fields shared by all fetcher threads Open

apache/stormcrawler#2101 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 14, 2026
SCProxy.toString() rebuilds the proxy URL with user and password, and the protocol logs it Open

apache/stormcrawler#2102 · 1 comment · 0 reactions · 1 assignee · Indexed Sep 14, 2026
OpenSearch addresses default to http and Basic credentials are registered for any host Open

apache/stormcrawler#2103 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Enable Storm dynamic producer batching in the archetype configurations Open

apache/stormcrawler#2131 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
fetcher.thread.timeout: cancel the okhttp call instead of interrupting a helper thread Open

apache/stormcrawler#2134 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026
Politeness keys do not collapse IDN host spellings (unicode vs punycode) Open

apache/stormcrawler#2145 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 14, 2026

apache/stormcrawler

Get new issues in your inbox