Azure Functions not dequeing messages under load

I have a function app running 1.0.13 of the .Net Functions SDK, deployed to a US North Central consumption based plan, configured for 2.0 beta. Under high loads, messages which have been dequeued more then the configured maxDequeueCount are not being moved to the poison queue. When I finally decided to stop my function so I could observe the queue, the first 32 messages had dequeue counts of 65-69 while the configured maxDequeueCount was 3.

When I first deployed my function, I had a bug, and observed the message getting moved to the poison queue as expected. After fixing that bug and running a few more tests with 1-10 messages, I enqueued 1000 messages to be worked. Each message results in anywhere from 100 to 5000 messages to be enqueued onto a second queue. The code for the 2nd queue will get a list of blob directory contents, and then enqueue up to 4 messages on a 3rd queue.

I am getting regular exceptions from the Health Monitor related to the Http Connnection Limit being exceeded. Other then that AppInsights is reporting that my "requests" are all succeeding.

269 of the 1000 messages never finished (for some reason I have yet to determine), and were continually retried until I stopped the function. 5 messages did get moved to the poison queue, but I have no idea what there dequeue count was prior to them finally being moved. Based on when they were inserted, I have to imagine that the dequeue count was > 3.

#### Investigative information

Please provide the following:

- Timestamp: 6/19/2018, 15:11:03 UTC
- Function App version: 2.0-beta
- Function App name:
- Function name(s) (as appropriate): Backup-LegacyMemberContainer-ProcessMember
- Invocation ID: fd504dcc-39da-4ecd-a05d-d3a5c30ee97a
- Region: North Central US
- Consumption Base Plan: Yes

#### Repro steps

See above. I have not tried to distill this down into a self contained repro case yet.

#### Expected behavior

Queue messages are moved to the poison queue when there dequeue count exceeds the maxDequeueCount at all times.

#### Actual behavior

Queue messages are showing a dequeue count of 65-69 while maxDequeueCount is set to 3. This seems to occur only under high load.

#### Known workarounds

None that I know of at this time.


Start with the reported Backup-LegacyMemberContainer-ProcessMember function, the maxDequeueCount setting, and the Health Monitor exceptions about the HTTP connection limit. Reproduce the high-load scenario described with 1,000 messages and inspect whether messages exceeding the configured count reach the poison queue. Done means the behavior is isolated and queue messages consistently move to the poison queue after the configured limit.

Written by the indexing model from the issue text.