redpanda-data / redpanda-data/connect
aws_kinesis input plugin unable to read aggregated records
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 8.8k
- Forks
- 969
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 64
Description
We use KPL to insert data into kinesis, and we use aggregated records which has compressed data. As per the logic, we store multiple user records in single kinesis record, and the data of the record will be compressed.
While trying to read this data, using aws_kinesis input plugin and enabling decompress processor, getting invalid gzip header error and decompression is not happening.
But when I ensure the aggregation is less during insertion, i.e. when no:of user records are equal to no:of kinesis records, not seeing any issue, with reading of data and decompression is happening correctly.
Is this the expected behaviour with aggregated kinesis records??
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the aws_kinesis input plugin and the decompress processor, then reproduce the difference between KPL aggregated records and one-user-per-Kinesis-record input. Check how aggregated records and their compression are interpreted, and use the observed invalid gzip header failure to define the expected behavior. Done means the supported behavior is clarified and aggregated records are handled or explicitly rejected with a clear test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100