Add JsonKinesisSource for DeltaStreamer ingestion from AWS Kinesis Data Streams
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Feature Description
**What the feature achieves:**
Adds support for ingesting data from AWS Kinesis Data Streams into Hudi tables via DeltaStreamer. Users can stream JSON records from Kinesis into Hudi with checkpointing, multi-shard reads, and incremental ingestion, in line with existing Kafka/DFS sources.
**Why this feature is needed:**
Kinesis is widely used for real-time streaming on AWS. Teams using Kinesis and Hudi currently need custom code or separate pipelines to move Kinesis data into the lakehouse. This creates duplication, inconsistency, and extra maintenance. Native Kinesis support in DeltaStreamer gives a single, well-tested path from Kinesis to Hudi, with checkpointing and support for upserts and other Hudi operations.
### User Experience
**How users will use this feature:**
Users configure DeltaStreamer with JsonKinesisSource and set Kinesis options (stream name, region, endpoint, starting position) in a properties file. They run DeltaStreamer as usual; it consumes records from Kinesis, writes them to a Hudi table using the chosen write operation (e.g., UPSERT), and tracks offsets for resumable, incremental ingestion across restarts.
### Hudi RFC Requirements
**RFC PR link:** (if applicable)
**Why RFC is/isn't needed:**
- Does this change public interfaces/APIs? (No)
- Does this change storage format? (No)
- Justification:
Small feature
Contributor guide
No contributing guide indexed for this repository
Research direction
Begin at DeltaStreamer and compare the existing Kafka and DFS sources to understand how source configuration, checkpointing, incremental ingestion, and write operations are handled. Confirm the required Kinesis properties and integration boundaries; done means JSON records can be ingested into Hudi with multi-shard reads and resumable offsets.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud, data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100