Enhancement: Make kinesis_streams MAX_EVENT_SIZE configurable or support >1MB records
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 58
Description
**Is your feature request related to a problem? Please describe.**
The plugin _out_kinesis_streams_ has a hard-coded maximum record size:
- MAX_EVENT_SIZE = 1048556
This causes records larger than 1MB to be discarted with a warning like:
- Discarding record which is larger than max size allowed by Firehose, %s
This behavior limits use cases where larger records are valid (especially since AWS Kinesis Data Streams now supports up to 10MB record sizes when the stream supports “large records”).
According to [AWS’s official announcement](https://aws.amazon.com/pt/blogs/big-data/amazon-kinesis-data-streams-now-supports-10x-larger-record-sizes-simplifying-real-time-data-processing/), Amazon Kinesis Data Streams now supports record sizes up to 10 MiB, a tenfold increase over the previous 1 MiB limit.
**Describe the solution you'd like**
Please consider the following improvement:
- Make the maximum record size configurable (e.g., via a plugin parameter), so that users can align the plugin behavior with the actual stream configuration and avoid discarding valid records.
**Describe alternatives you've considered**
Please consider the following improvement:
- Make the maximum record size bigger, so that users can avoid discarding valid records.
**Additional context**
N/A
Contributor guide
Research direction
Start by locating the out_kinesis_streams plugin and its MAX_EVENT_SIZE definition, then inspect how plugin parameters are declared and how oversized records reach the discard warning. Check the relevant plugin tests, if present. Done means the maximum size can follow the Kinesis stream configuration or valid records above 1 MiB are supported without being discarded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, c
- Domain
- data, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100