Azure function app crashes and restarts very often

Our team ships a V2 Azure function app (with around 7-8) functions in it. The primary job of the function app is to act as a data processing pipeline where one steps executes and triggers another via queue based triggers. There are also a couple of functions that get triggered via timers. And there are a couple of functions that are decorated with the `Singleton` attribute.

Each of our customers gets their own deployment on their Azure tenant.

We've noticed that while the system works reasonably well for a small number of users, it starts crashing as the processing burden increases. And I don't mean to say that the processing burden is huge. We start seeing the issue fairly easily at relatively low user counts (less than 100).

We've been around the block and addressed a few different issues over the last few months (memory consumption, SNAT port exhaustion, TCP connections etc.) to improve the stability of our app, and think this particular instability is because of storage account issues, that is, related to the queues and singleton/timer locks used by the Azure functions host/infrastructure. We've been looking at logs on Kudu. eventlogs.xml is clean, the host log file occasionally will have an exception like this:

Singleton lock renewal failed for blob 'FUNCTION_APP_ID/FUNCTION_APP_NAME'
with error code 409: LeaseIdMismatchWithLeaseOperation. The last successful renewal completed at 2020-07-
06T18:00:39.889Z (7524 milliseconds ago) with a duration of 28948 milliseconds. The lease period was 15000 milliseconds.

On researching the issue, it seems like it is not uncommon, and one explanation is that the app service plan is max'd out and the host is getting starved and unable to get execution time and hence is leading to, for example, a lease timeout when interacting with the storage account.

What makes this issue even fuzzier is that while we see such errors in the logs rarely, the function app itself is restarting many times. So we don't know what other problems are occurring, or why not all problems are being captured in the logs. We have set the log level to "Trace".

Happy to share more information as needed. This is causing a huge issue for us in production and we would appreciate any help or pointers toward a solution.