Azure / Azure/azure-functions-host

Is there any way to have more control over batch sizes for Event Hub triggered functions?

Open
#7,268 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

#### Is your question related to a specific version? If so, please specify:

No

#### What language does your question apply to? (e.g. C#, JavaScript, Java, All)

All

#### Question

We have multiple Function Apps with Event Hub triggered functions that are consuming from Event Hubs that have a constant incoming message rate of > 100k per minute. Our `function.json` file has `cardinality` set to `many` and the `host.json` file has `maxBatchSize` set to `256`. Most of our function invocations are only receiving one event in the batch. This is causing us problems because the downstream service that we send the events to can be a bottleneck but we can mitigate this if we send multiple events in a batch. We cannot do this currently because we are mostly receiving one event at a time. Also, constantly receiving batch sizes of one is driving up the cost of running this solution. This is causing an excessive amount of transactions against the Storage Account (90k per minute which is over $250 a day when advanced threat protection is on). We can control this by setting `batchCheckpointFrequency` to a value > 75 but I would prefer to not have to set this so high (don't want to have to reprocess duplicate events if checkpointing fails). Since the pricing for Function Apps is based on the number of executions, batch sizes of one are driving up that cost category also.

Is there any way I can have more control over the number of events in a batch? I understand that `maxBatchSize` is not a guarantee of how many I will receive per invocation. We are looking to deploy thousands of Function Apps across many subscriptions so ideally I would like a solution that will work well across all of these. I've been told that I could possibly limit the scale out for the functions which would eventually force larger batch sizes per invocation to keep up with the incoming messages but I would prefer not to do this. We do not have any control over the incoming message rate to the Event Hub so we need the benefit of scale.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the Event Hub settings described in function.json and host.json, especially cardinality, maxBatchSize, and batchCheckpointFrequency. Trace how these settings affect invocation batching and determine whether a supported control for minimum or target batch size exists; done means documenting the available behavior or defining the required change.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.