Enable Storm dynamic producer batching in the archetype configurations
- Dominant language
- Java
- Stars
- 995
- Forks
- 292
- Avg merge
- 2d 49m
- Merged PRs (30d)
- 62
Description
The `crawler-conf.yaml` generated by the archetypes leaves Storm's inter-executor batching at its default (`topology.producer.batch.size: 1`), so no batching happens between the spout, fetcher, parser, indexer and status updater executors.
Storm 3.0 adds `topology.producer.batch.dynamic`: when enabled together with a `topology.producer.batch.size` greater than 1, the effective batch size adapts at runtime between 1 and the configured size with AIMD, staying small under light load to keep latency low and growing under sustained load to preserve throughput. That removes the usual objection to batching in a crawl topology, where traffic is bursty and latency to the status updater matters.
Proposal: enable it in the archetype configurations with a conservative batch size below `topology.max.spout.pending`, and document the Storm topology settings that affect a crawl (`topology.message.timeout.secs`, `topology.max.spout.pending`, the producer batch settings and the flush interval), which are currently not covered in the configuration documentation.
Contributor guide
Research direction
Inspect the archetype-generated crawler-conf.yaml files and the configuration documentation first, then verify the Storm 3.0 producer batch settings and their relationship to topology.max.spout.pending. Done means the archetypes use a conservative dynamic batch configuration and the documentation covers message timeout, max spout pending, producer batching, and flush interval settings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, documentation
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100