Azure / Azure/azure-functions-host
Processing delays when scaling out Event Hub triggered function
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
#### Repro steps
1. Create Cosmos DB with Document DB API (and a collection with enough throughput so that the DB is not the bottleneck)
2. Create Event Hub and a consumer group the function will use
3. Create a Node.js function that uses the created Event Hub as trigger as the Cosmos DB as the output (see https://gist.github.com/vjrantal/1d5d3361bf6f76a2c698c3104b4262ec for the `function.json` and `index.js` files I used in my test)
4. Start bombarding the Event Hub with messages and observe how Azure Functions scales out the processing (I used Apache Benchmark to post 1 million events to the hub by using round-robin model to distribute events evenly to the allocated Event Hub partitions)
#### Expected behavior
When the runtime determines scaling out is required, it happens without significant delays in event processing.
#### Actual behavior
As seen in below screenshot, there seems to be a gap in event processing when the initial scale out occurs. It also seems like the per request duration slows down significantly after the scale out (e.g., 50 percentile duration before scale out is something like ~18 ms, but afterwards something like ~2167 ms).

#### Related information
I run the test multiple times and tested with cardinality many instead of one. I also tested without the Document DB output. Furthermore, the amount of Event Hub partitions did not seem to matter (tried with 4 and 32). In all 5 trials, I was able to reproduce similar behavior.
During the gap in event processing, I can see that some errors occur, which are something I think I could not catch in my Node.js "user code":

UTC timestamps are in the screenshots and the Function App is called `documentdb-function-1496656123610`. All resources I created were in North Europe region. I had not touched `host.json` so everything there were the default values.
Contributor guide
Assessment
This issue has not been assessed yet.