apache / apache/hudi

Add JsonKinesisSource for DeltaStreamer ingestion from AWS Kinesis Data Streams

Open
#18,228 0 comments 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Feature Description

**What the feature achieves:**
Adds support for ingesting data from AWS Kinesis Data Streams into Hudi tables via DeltaStreamer. Users can stream JSON records from Kinesis into Hudi with checkpointing, multi-shard reads, and incremental ingestion, in line with existing Kafka/DFS sources.

**Why this feature is needed:**
Kinesis is widely used for real-time streaming on AWS. Teams using Kinesis and Hudi currently need custom code or separate pipelines to move Kinesis data into the lakehouse. This creates duplication, inconsistency, and extra maintenance. Native Kinesis support in DeltaStreamer gives a single, well-tested path from Kinesis to Hudi, with checkpointing and support for upserts and other Hudi operations.

### User Experience

**How users will use this feature:**
Users configure DeltaStreamer with JsonKinesisSource and set Kinesis options (stream name, region, endpoint, starting position) in a properties file. They run DeltaStreamer as usual; it consumes records from Kinesis, writes them to a Hudi table using the chosen write operation (e.g., UPSERT), and tracks offsets for resumable, incremental ingestion across restarts.

### Hudi RFC Requirements

**RFC PR link:** (if applicable)

**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (No)
- Does this change storage format? (No)
- Justification:
Small feature

Contributor guide

No contributing guide indexed for this repository

Research direction

Begin at DeltaStreamer and compare the existing Kafka and DFS sources to understand how source configuration, checkpointing, incremental ingestion, and write operations are handled. Confirm the required Kinesis properties and integration boundaries; done means JSON records can be ingested into Hudi with multi-shard reads and resumable offsets.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
cloud, data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.