apache / apache/druid

Amazon Kinesis Support Enhanced Fan-Out

Open
#15,800 5 comments 0 reactions 0 assignees View on GitHub
Feature/Change Description
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description
Feel free to close this if there is a better place to put this or if I'm incorrect. But I feel like there are improvements that could be made to the kinesis ingestion to resolve the main "known issue" under the kinesis druid docs.

> Before you deploy the Kinesis extension to production, consider the following known issues:
Avoid implementing more than one Kinesis supervisor that reads from the same Kinesis stream for ingestion. Kinesis has a per-shard read throughput limit and having multiple supervisors on the same stream can reduce available read throughput for an individual supervisor's tasks.

What does Enhanced fan out do?
> A consumer that uses enhanced fan-out doesn't have to contend with other consumers who are receiving data from the stream

Source Docs:
- https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/
- V1 docs: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/AmazonKinesis.html#putRecords-com.amazonaws.services.kinesis.model.PutRecordsRequest-
-

https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html

---

I have not dug into the kinesis consuming code enough to say what the lift would be to support this, but thought posting this for visibility might be worth while? :shrug:

Contributor guide

Open the contributing guide

Research direction

Start by tracing Druid's Kinesis ingestion code and reading the linked AWS Enhanced Fan-Out documentation, including the enhanced consumers API. Define the required consumer changes and tests before implementing support, with completion shown by Kinesis ingestion using enhanced fan-out without the documented shared-stream read-throughput limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.