aws-samples / aws-samples/integration-sample-lambda-msk

Parallelization Factor for MSK Event Source Mapping

Open
#2 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
11
Forks
10
Avg merge
1d 1h
Merged PRs (30d)
1

Description

Hi,

Is there a way to set concurrency of the lambda function so that multiple batches can be processed simultaneouly?

From the [aws lambda cli docs](https://docs.aws.amazon.com/cli/latest/reference/lambda/create-event-source-mapping.html) I see there is a `--parallelization-factor` which I think is supported for Kinesis streams, but not MSK?

**Are there any plans to support this for MSK?**

If I understood the MSK event source correctly, it polls a target MSK topic across partitions and invokes a lambda function with the obtained batch synchronously? So in order to poll for another batch, the invoked lambda function must complete execution before a subsequent poll can begin?

Therefore, the throughput that can be handled is a function of:
- The number of topic partitions (more partitions allows for higher throughput)
- The batch size of the MSK event source (10,000 is currently the max)
- The execution time of the lambda function.

> For example, if the execution takes 100ms vs. 10secs this makes a difference to the consumer lag in high-throughput topics?

Many thanks,

David

Contributor guide

Open the contributing guide

Research direction

No repository files, tests, or implementation entry points are mentioned. Start by reviewing the AWS Lambda MSK event-source documentation and the repository structure to determine whether this is implementable here; done would require a confirmed scope and an agreed way to support or document parallel processing.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.