redpanda-data / redpanda-data/connect

aws_kinesis input plugin unable to read aggregated records

Open
#1,683 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

inputs needs investigation ux
Dominant language
Go
Stars
8.8k
Forks
969
Avg merge
1d 13h
Merged PRs (30d)
64

Description

We use KPL to insert data into kinesis, and we use aggregated records which has compressed data. As per the logic, we store multiple user records in single kinesis record, and the data of the record will be compressed.
While trying to read this data, using aws_kinesis input plugin and enabling decompress processor, getting invalid gzip header error and decompression is not happening.

But when I ensure the aggregation is less during insertion, i.e. when no:of user records are equal to no:of kinesis records, not seeing any issue, with reading of data and decompression is happening correctly.

Is this the expected behaviour with aggregated kinesis records??

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the aws_kinesis input plugin and the decompress processor, then reproduce the difference between KPL aggregated records and one-user-per-Kinesis-record input. Check how aggregated records and their compression are interpreted, and use the observed invalid gzip header failure to define the expected behavior. Done means the supported behavior is clarified and aggregated records are handled or explicitly rejected with a clear test.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.