apache / apache/druid

Abstract away BlockingQueue.offer()/poll() which notify after some period of inability to proceed

Open
#7,037 2 comments 0 reactions 0 assignees View on GitHub
Area - Operations Refactoring stale
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

There are several places in Druid code which implement some sort of untimed `BlockingQueue.take()`, but they want to notify or log about inability to make progress, so they use `BlockingQueue.offer()`:

https://github.com/apache/incubator-druid/blob/fafbc4a80e0bba0c1fbb5066b9f0ce37ffcab929/extensions-contrib/ambari-metrics-emitter/src/main/java/org/apache/druid/emitter/ambari/metrics/AmbariMetricsEmitter.java#L115-L127

https://github.com/apache/incubator-druid/blob/fafbc4a80e0bba0c1fbb5066b9f0ce37ffcab929/examples/src/main/java/org/apache/druid/examples/twitter/TwitterSpritzerFirehoseFactory.java#L165-L168

https://github.com/apache/incubator-druid/blob/fafbc4a80e0bba0c1fbb5066b9f0ce37ffcab929/extensions-core/kinesis-indexing-service/src/main/java/org/apache/druid/indexing/kinesis/KinesisRecordSupplier.java#L176-L179

https://github.com/apache/incubator-druid/blob/fafbc4a80e0bba0c1fbb5066b9f0ce37ffcab929/server/src/main/java/org/apache/druid/segment/realtime/firehose/EventReceiverFirehoseFactory.java#L376-L383

https://github.com/apache/incubator-druid/blob/fafbc4a80e0bba0c1fbb5066b9f0ce37ffcab929/extensions-contrib/graphite-emitter/src/main/java/org/apache/druid/emitter/graphite/GraphiteEmitter.java#L109-L121

They all use different strategies and patterns. It might be reasonable to extract this pattern as some sort of utility, or e. g. specify a common "timeout" in druid after which such events are logged, or a configuration might be added to choose between logging and emitting such events as alerts.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the queue handling in AmbariMetricsEmitter.java, TwitterSpritzerFirehoseFactory.java, KinesisRecordSupplier.java, EventReceiverFirehoseFactory.java, and GraphiteEmitter.java. Identify the common progress-notification behavior and decide whether a utility, shared timeout, or configurable alerting approach is intended; done requires a clearly defined design applied consistently across the listed entry points.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.