aws-samples / aws-samples/integration-sample-lambda-msk

Parallelization Factor for MSK Event Source Mapping

未关闭
#2 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Java
星标
11
派生
10
平均合并
1 天 1 小时
30 天内合并 PR
1

描述

Hi,

Is there a way to set concurrency of the lambda function so that multiple batches can be processed simultaneouly?

From the [aws lambda cli docs](https://docs.aws.amazon.com/cli/latest/reference/lambda/create-event-source-mapping.html) I see there is a `--parallelization-factor` which I think is supported for Kinesis streams, but not MSK?

**Are there any plans to support this for MSK?**

If I understood the MSK event source correctly, it polls a target MSK topic across partitions and invokes a lambda function with the obtained batch synchronously? So in order to poll for another batch, the invoked lambda function must complete execution before a subsequent poll can begin?

Therefore, the throughput that can be handled is a function of:
- The number of topic partitions (more partitions allows for higher throughput)
- The batch size of the MSK event source (10,000 is currently the max)
- The execution time of the lambda function.

> For example, if the execution takes 100ms vs. 10secs this makes a difference to the consumer lag in high-throughput topics?

Many thanks,

David

贡献指南

打开贡献指南

调研方向

No repository files, tests, or implementation entry points are mentioned. Start by reviewing the AWS Lambda MSK event-source documentation and the repository structure to determine whether this is implementable here; done would require a confirmed scope and an agreed way to support or document parallel processing.

由索引模型根据 Issue 内容生成。

评估

技术栈
aws
领域
cloud
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
需要澄清
新手友好度
20/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。