elastic / elastic/beats

Kafka Output Bulk Flush Configuration Enhancement

Open
#48,503 1 comment 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

The current Kafka output implementation in Beats lacks essential batch control parameters, specifically `Flush.Bytes` and `Flush.Messages` from the sarama client. This limitation causes performance issues in production environments:

Current Implementation
```go
// libbeat/outputs/kafka/config.go#newSaramaConfig()
// Only configures MaxMessages and Frequency
k.Producer.Flush.MaxMessages = config.BulkMaxSize
if config.BulkFlushFrequency > 0 {
k.Producer.Flush.Frequency = config.BulkFlushFrequency
}
```

## Issues Identified

- Small Batch Requests: No byte-based or message-count-based triggers cause frequent small requests
- High Kafka CPU Usage: Many small batches increase broker processing overhead

I would like to add two new configuration options to expose sarama's Flush.Bytes and Flush.Messages.

Contributor guide

Open the contributing guide

Research direction

Start in libbeat/outputs/kafka/config.go, especially newSaramaConfig(), and inspect how the existing bulk options are defined and passed into the Sarama producer configuration. Add configuration paths for Flush.Bytes and Flush.Messages, then verify that the Kafka output exposes both controls alongside the existing MaxMessages and Frequency settings.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.