awslabs / awslabs/amazon-kinesis-client-python

Using Python KCL library for mutli-stream processing

Open
#172 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
377
Forks
228
PR merge metrics
No merged PRs in 30d

Description

I am using `amazon-kclpy==2.0.6` to process kinesis streams. This works well for single stream processing. From the Java library, I see that multi-stream support was introduced in the [java library version 2.3.0](https://github.com/awslabs/amazon-kinesis-client#release-230-august-17-2020). Also worth noting that `amazon-kclpy v2.0.6` uses Amazon Kinesis Client v2.3.9 [as dependency](https://github.com/awslabs/amazon-kinesis-client-python#release-206-november-23-2021).

The [official docs](https://aws.amazon.com/about-aws/whats-new/2020/10/kinesis-client-library-enables-multi-stream-processing/#:~:text=KCL%20enables%20you%20to%20focus,processing%20data%20with%20fault%2Dtolerance.) released with the Java library suggests
> With this new capability, you can update the list of streams at runtime for multi-stream processing in a scalable KCL application without redeploying the application

My use case requires one application to listen/process multiple Kinesis streams. As the stream names are static, could even provide the stream names as part of the configuration itself.

The issue:

As the python client interacts with the Java Multilangdaemon, we have the ability to provide the [configuration file to the MultilangDaemon](https://github.com/awslabs/amazon-kinesis-client-python/blob/master/samples/amazon_kclpy_helper.py#L124). Yet :
1. We cannot provide multiple stream names ( a list of names) as the `streamName` in properties file. This field is expected to be a string.
2. We cannot provide two properties file ( with each file set up with different `streamName` to the Multilangdaemon.

How can we configure the KCL Python to use multiple stream? Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.