elastic / elastic/logstash

Add byte-based flush setting for ES

Open
#4,000 3 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

Currently `flush_size` is the only knob to control how much data a worker sends to ES, but if data varies in size, this can range from annoying (inconsistent heap use) to catastrophic on a low-margin system.

There should be some kind of `flush_bytes` setting as well, where each time a message is slated to be added to a pending bulk request the total bytes of the docs are considered.

Contributor guide

Open the contributing guide

Research direction

Start by locating the worker's existing flush_size handling and the code that builds pending bulk requests for Elasticsearch. Trace how document sizes can be totaled when a message is added, then determine the expected interaction between flush_bytes and flush_size. Done means a configurable byte limit controls pending bulk request size without removing the existing message-count setting.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.