[SUPPORT] Reading data from Kinesis data stream and storing into S3 using Hudi for KDA-Flink
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Hey guys, I am trying to implement reading from kinesis data streams and storing it into an s3 bucket using Hudi.
I was able to add the data into s3 by referring and running the following code- https://github.com/awsalialem/amazon-kinesis-data-analytics-java-examples/blob/master/S3Sink/src/main/java/com/amazonaws/services/kinesisanalytics/S3StreamingSinkJob.java
I wanted to know how I can further store the data into S3 using Hudi connector while reading from kinesis data streams. Is there any modification I can do to the code?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the referenced S3StreamingSinkJob.java example and the KDA-Flink Hudi connector integration described in the issue. Determine the supported way to read from Kinesis and write Hudi data to S3, then verify that the resulting guidance covers the requested end-to-end setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- cloud, data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100