Kafka Output Bulk Flush Configuration Enhancement
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
The current Kafka output implementation in Beats lacks essential batch control parameters, specifically `Flush.Bytes` and `Flush.Messages` from the sarama client. This limitation causes performance issues in production environments:
Current Implementation
```go
// libbeat/outputs/kafka/config.go#newSaramaConfig()
// Only configures MaxMessages and Frequency
k.Producer.Flush.MaxMessages = config.BulkMaxSize
if config.BulkFlushFrequency > 0 {
k.Producer.Flush.Frequency = config.BulkFlushFrequency
}
```
## Issues Identified
- Small Batch Requests: No byte-based or message-count-based triggers cause frequent small requests
- High Kafka CPU Usage: Many small batches increase broker processing overhead
I would like to add two new configuration options to expose sarama's Flush.Bytes and Flush.Messages.
Contributor guide
Research direction
Start in libbeat/outputs/kafka/config.go, especially newSaramaConfig(), and inspect how the existing bulk options are defined and passed into the Sarama producer configuration. Add configuration paths for Flush.Bytes and Flush.Messages, then verify that the Kafka output exposes both controls alongside the existing MaxMessages and Frequency settings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100