Make asynchronous emitters have the same (configurable?) policy when event queue is full
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
This issue narrows the idea of #7037.
There are many emitters (at least: `AmbariMetricsEmitter`, `GraphiteEmitter`, `StatsDEmitter` and `KafkaEmitter` and `HttpPostEmitter`, I haven't checked other) that use the same producer-consumer pattern for asynchronous emit: `emit()` pushes the event to some queue (one of the queues), and there is an asynchronous executor that retrieves events from the queue and sends them over network to emit.
`AmbariMetricsEmitter` and `GraphiteEmitter` use the same policy when the queue is full (they log a warning). But `StatsDEmitter` apparently silently discards new events when the queue is full (see `NonBlockingStatsDClient` code). `KafkaEmitter` discards new events, but increments "lost events" counts. `HttpPostEmitter` packs events in batches and drops the oldest batch when overwhelmed, simultaneously logging that (see `HttpPostEmitter.limitBuffersToEmitSize()` and `limitFailedBuffersSize()`).
I think all emitters should be similar in this regard. Probably event throttling policy should be configurable.
Related to #2868.
Contributor guide
Research direction
Compare the queue-full behavior in AmbariMetricsEmitter, GraphiteEmitter, StatsDEmitter and NonBlockingStatsDClient, KafkaEmitter, and HttpPostEmitter, including HttpPostEmitter.limitBuffersToEmitSize() and limitFailedBuffersSize(). First map what each emitter currently logs, drops, or counts; done means the emitters follow one agreed policy, with configurability if that design is retained, and the differing outcomes are covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100