apache / apache/hudi

[SUPPORT] Reading data from Kinesis data stream and storing into S3 using Hudi for KDA-Flink

Open
#6,475 6 comments 0 reactions 0 assignees View on GitHub
engine:flink priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Hey guys, I am trying to implement reading from kinesis data streams and storing it into an s3 bucket using Hudi.
I was able to add the data into s3 by referring and running the following code- https://github.com/awsalialem/amazon-kinesis-data-analytics-java-examples/blob/master/S3Sink/src/main/java/com/amazonaws/services/kinesisanalytics/S3StreamingSinkJob.java

I wanted to know how I can further store the data into S3 using Hudi connector while reading from kinesis data streams. Is there any modification I can do to the code?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the referenced S3StreamingSinkJob.java example and the KDA-Flink Hudi connector integration described in the issue. Determine the supported way to read from Kinesis and write Hudi data to S3, then verify that the resulting guidance covers the requested end-to-end setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cloud, data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.