Add byte-based flush setting for ES
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
Currently `flush_size` is the only knob to control how much data a worker sends to ES, but if data varies in size, this can range from annoying (inconsistent heap use) to catastrophic on a low-margin system.
There should be some kind of `flush_bytes` setting as well, where each time a message is slated to be added to a pending bulk request the total bytes of the docs are considered.
Contributor guide
Research direction
Start by locating the worker's existing flush_size handling and the code that builds pending bulk requests for Elasticsearch. Trace how document sizes can be totaled when a message is added, then determine the expected interaction between flush_bytes and flush_size. Done means a configurable byte limit controls pending bulk request size without removing the existing message-count setting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100