aws-samples / aws-samples/integration-sample-lambda-msk
Parallelization Factor for MSK Event Source Mapping
- Ngôn ngữ chính
- Java
- Star
- 11
- Fork
- 10
- Merge trung bình
- 1 ngày 1 giờ
- Pull request đã merge (30 ngày)
- 1
Mô tả
Hi,
Is there a way to set concurrency of the lambda function so that multiple batches can be processed simultaneouly?
From the [aws lambda cli docs](https://docs.aws.amazon.com/cli/latest/reference/lambda/create-event-source-mapping.html) I see there is a `--parallelization-factor` which I think is supported for Kinesis streams, but not MSK?
**Are there any plans to support this for MSK?**
If I understood the MSK event source correctly, it polls a target MSK topic across partitions and invokes a lambda function with the obtained batch synchronously? So in order to poll for another batch, the invoked lambda function must complete execution before a subsequent poll can begin?
Therefore, the throughput that can be handled is a function of:
- The number of topic partitions (more partitions allows for higher throughput)
- The batch size of the MSK event source (10,000 is currently the max)
- The execution time of the lambda function.
> For example, if the execution takes 100ms vs. 10secs this makes a difference to the consumer lag in high-throughput topics?
Many thanks,
David
Hướng dẫn đóng góp
Hướng nghiên cứu
No repository files, tests, or implementation entry points are mentioned. Start by reviewing the AWS Lambda MSK event-source documentation and the repository structure to determine whether this is implementable here; done would require a confirmed scope and an agreed way to support or document parallel processing.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- aws
- Lĩnh vực
- cloud
- Loại issue
- Tính năng
- Độ khó
- 5/5
- Thời gian dự kiến
- Hơn một tuần
- Mức độ hoạt động
- Đình trệ
- Độ rõ ràng
- Cần làm rõ
- Mức phù hợp với người mới
- 20/100