Azure / Azure/azure-functions-host

Processing delays when scaling out Event Hub triggered function

Open
#1,601 10 comments 0 reactions 1 assignee Claimed by @paulbatum View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

#### Repro steps

1. Create Cosmos DB with Document DB API (and a collection with enough throughput so that the DB is not the bottleneck)

2. Create Event Hub and a consumer group the function will use

3. Create a Node.js function that uses the created Event Hub as trigger as the Cosmos DB as the output (see https://gist.github.com/vjrantal/1d5d3361bf6f76a2c698c3104b4262ec for the `function.json` and `index.js` files I used in my test)

4. Start bombarding the Event Hub with messages and observe how Azure Functions scales out the processing (I used Apache Benchmark to post 1 million events to the hub by using round-robin model to distribute events evenly to the allocated Event Hub partitions)

#### Expected behavior

When the runtime determines scaling out is required, it happens without significant delays in event processing.

#### Actual behavior

As seen in below screenshot, there seems to be a gap in event processing when the initial scale out occurs. It also seems like the per request duration slows down significantly after the scale out (e.g., 50 percentile duration before scale out is something like ~18 ms, but afterwards something like ~2167 ms).

functions-scaling

#### Related information

I run the test multiple times and tested with cardinality many instead of one. I also tested without the Document DB output. Furthermore, the amount of Event Hub partitions did not seem to matter (tried with 4 and 32). In all 5 trials, I was able to reproduce similar behavior.

During the gap in event processing, I can see that some errors occur, which are something I think I could not catch in my Node.js "user code":

functions-error

UTC timestamps are in the screenshots and the Function App is called `documentdb-function-1496656123610`. All resources I created were in North Europe region. I had not touched `host.json` so everything there were the default values.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.