Azure / Azure/azure-functions-host
Is there any way to have more control over batch sizes for Event Hub triggered functions?
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
#### Is your question related to a specific version? If so, please specify:
No
#### What language does your question apply to? (e.g. C#, JavaScript, Java, All)
All
#### Question
We have multiple Function Apps with Event Hub triggered functions that are consuming from Event Hubs that have a constant incoming message rate of > 100k per minute. Our `function.json` file has `cardinality` set to `many` and the `host.json` file has `maxBatchSize` set to `256`. Most of our function invocations are only receiving one event in the batch. This is causing us problems because the downstream service that we send the events to can be a bottleneck but we can mitigate this if we send multiple events in a batch. We cannot do this currently because we are mostly receiving one event at a time. Also, constantly receiving batch sizes of one is driving up the cost of running this solution. This is causing an excessive amount of transactions against the Storage Account (90k per minute which is over $250 a day when advanced threat protection is on). We can control this by setting `batchCheckpointFrequency` to a value > 75 but I would prefer to not have to set this so high (don't want to have to reprocess duplicate events if checkpointing fails). Since the pricing for Function Apps is based on the number of executions, batch sizes of one are driving up that cost category also.
Is there any way I can have more control over the number of events in a batch? I understand that `maxBatchSize` is not a guarantee of how many I will receive per invocation. We are looking to deploy thousands of Function Apps across many subscriptions so ideally I would like a solution that will work well across all of these. I've been told that I could possibly limit the scale out for the functions which would eventually force larger batch sizes per invocation to keep up with the incoming messages but I would prefer not to do this. We do not have any control over the incoming message rate to the Event Hub so we need the benefit of scale.
Contributor guide
Research direction
Start by reviewing the Event Hub settings described in function.json and host.json, especially cardinality, maxBatchSize, and batchCheckpointFrequency. Trace how these settings affect invocation batching and determine whether a supported control for minimum or target batch size exists; done means documenting the available behavior or defining the required change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100