apache / apache/druid

Make asynchronous emitters have the same (configurable?) policy when event queue is full

Open
#7,057 11 comments 0 reactions 0 assignees View on GitHub
Area - Metrics/Event Emitting Area - Operations Contributions Welcome stale
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

This issue narrows the idea of #7037.

There are many emitters (at least: `AmbariMetricsEmitter`, `GraphiteEmitter`, `StatsDEmitter` and `KafkaEmitter` and `HttpPostEmitter`, I haven't checked other) that use the same producer-consumer pattern for asynchronous emit: `emit()` pushes the event to some queue (one of the queues), and there is an asynchronous executor that retrieves events from the queue and sends them over network to emit.

`AmbariMetricsEmitter` and `GraphiteEmitter` use the same policy when the queue is full (they log a warning). But `StatsDEmitter` apparently silently discards new events when the queue is full (see `NonBlockingStatsDClient` code). `KafkaEmitter` discards new events, but increments "lost events" counts. `HttpPostEmitter` packs events in batches and drops the oldest batch when overwhelmed, simultaneously logging that (see `HttpPostEmitter.limitBuffersToEmitSize()` and `limitFailedBuffersSize()`).

I think all emitters should be similar in this regard. Probably event throttling policy should be configurable.

Related to #2868.

Contributor guide

Open the contributing guide

Research direction

Compare the queue-full behavior in AmbariMetricsEmitter, GraphiteEmitter, StatsDEmitter and NonBlockingStatsDClient, KafkaEmitter, and HttpPostEmitter, including HttpPostEmitter.limitBuffersToEmitSize() and limitFailedBuffersSize(). First map what each emitter currently logs, drops, or counts; done means the emitters follow one agreed policy, with configurability if that design is retained, and the differing outcomes are covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.