awslabs / awslabs/amazon-kinesis-client-python
Using Python KCL library for mutli-stream processing
- Dominant language
- Python
- Stars
- 377
- Forks
- 228
- PR merge metrics
- No merged PRs in 30d
Description
I am using `amazon-kclpy==2.0.6` to process kinesis streams. This works well for single stream processing. From the Java library, I see that multi-stream support was introduced in the [java library version 2.3.0](https://github.com/awslabs/amazon-kinesis-client#release-230-august-17-2020). Also worth noting that `amazon-kclpy v2.0.6` uses Amazon Kinesis Client v2.3.9 [as dependency](https://github.com/awslabs/amazon-kinesis-client-python#release-206-november-23-2021).
The [official docs](https://aws.amazon.com/about-aws/whats-new/2020/10/kinesis-client-library-enables-multi-stream-processing/#:~:text=KCL%20enables%20you%20to%20focus,processing%20data%20with%20fault%2Dtolerance.) released with the Java library suggests
> With this new capability, you can update the list of streams at runtime for multi-stream processing in a scalable KCL application without redeploying the application
My use case requires one application to listen/process multiple Kinesis streams. As the stream names are static, could even provide the stream names as part of the configuration itself.
The issue:
As the python client interacts with the Java Multilangdaemon, we have the ability to provide the [configuration file to the MultilangDaemon](https://github.com/awslabs/amazon-kinesis-client-python/blob/master/samples/amazon_kclpy_helper.py#L124). Yet :
1. We cannot provide multiple stream names ( a list of names) as the `streamName` in properties file. This field is expected to be a string.
2. We cannot provide two properties file ( with each file set up with different `streamName` to the Multilangdaemon.
How can we configure the KCL Python to use multiple stream? Thank you!
Contributor guide
Assessment
This issue has not been assessed yet.