Amazon Kinesis Support Enhanced Fan-Out
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Description
Feel free to close this if there is a better place to put this or if I'm incorrect. But I feel like there are improvements that could be made to the kinesis ingestion to resolve the main "known issue" under the kinesis druid docs.
> Before you deploy the Kinesis extension to production, consider the following known issues:
Avoid implementing more than one Kinesis supervisor that reads from the same Kinesis stream for ingestion. Kinesis has a per-shard read throughput limit and having multiple supervisors on the same stream can reduce available read throughput for an individual supervisor's tasks.
What does Enhanced fan out do?
> A consumer that uses enhanced fan-out doesn't have to contend with other consumers who are receiving data from the stream
Source Docs:
- https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/
- V1 docs: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/AmazonKinesis.html#putRecords-com.amazonaws.services.kinesis.model.PutRecordsRequest-
-
https://docs.aws.amazon.com/streams/latest/dev/building-enhanced-consumers-api.html
---
I have not dug into the kinesis consuming code enough to say what the lift would be to support this, but thought posting this for visibility might be worth while? :shrug:
Contributor guide
Research direction
Start by tracing Druid's Kinesis ingestion code and reading the linked AWS Enhanced Fan-Out documentation, including the enhanced consumers API. Define the required consumer changes and tests before implementing support, with completion shown by Kinesis ingestion using enhanced fan-out without the documented shared-stream read-throughput limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100