opensearch-project / opensearch-project/data-prepper
Support ByteCount for all byte-based configurations
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
Some Data Prepper configurations accept bytes values in the Data Prepper byte count, however some still take values in other formats. For example, the opensearch sink bulk_size accepts an integer in megabytes.
As a pipeline author or Data Prepper administrator I am not sure what value to use.
Describe the solution you'd like
Use the Data Prepper byte count format consistently for any time duration.
Existing configurations may need to be replaced so some of these changes may be breaking.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inventorying Data Prepper configurations that accept byte-based values, including the OpenSearch sink's bulk_size, and compare their current formats with the existing Data Prepper byte count format. Define the affected settings and compatibility implications; the work is done when byte-based configurations consistently use the byte count format.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100