apache / apache/stormcrawler

Enable Storm dynamic producer batching in the archetype configurations

Open
#2,131 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
995
Forks
292
Avg merge
2d 49m
Merged PRs (30d)
62

Description

The `crawler-conf.yaml` generated by the archetypes leaves Storm's inter-executor batching at its default (`topology.producer.batch.size: 1`), so no batching happens between the spout, fetcher, parser, indexer and status updater executors.

Storm 3.0 adds `topology.producer.batch.dynamic`: when enabled together with a `topology.producer.batch.size` greater than 1, the effective batch size adapts at runtime between 1 and the configured size with AIMD, staying small under light load to keep latency low and growing under sustained load to preserve throughput. That removes the usual objection to batching in a crawl topology, where traffic is bursty and latency to the status updater matters.

Proposal: enable it in the archetype configurations with a conservative batch size below `topology.max.spout.pending`, and document the Storm topology settings that affect a crawl (`topology.message.timeout.secs`, `topology.max.spout.pending`, the producer batch settings and the flush interval), which are currently not covered in the configuration documentation.

Contributor guide

Open the contributing guide

Research direction

Inspect the archetype-generated crawler-conf.yaml files and the configuration documentation first, then verify the Storm 3.0 producer batch settings and their relationship to topology.max.spout.pending. Done means the archetypes use a conservative dynamic batch configuration and the documentation covers message timeout, max spout pending, producer batching, and flush interval settings.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, documentation
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.