Add a Blob storage implementation for IDebugStateProvider

As described in issue https://github.com/Azure/azure-functions-host/issues/3463, our mechanism of conditionally enabling “debug mode” in the Functions runtime when the portal is connected doesn’t work. As review, this is how it is supposed to work:

- Portal makes requests to /admin/* routes to invoke functions, get host status, etc.
- Some of these routes trigger the host to enter a “debug mode” by touching a debug_sentinel file in the Logs\Host directory
- All host instances have file change notifications set up on the debug_sentinel file – when they see it updated, they enter a debug mode with a 15 minute timeout
- this relies on a shared file system. We’re leveraging notifications on a shared file to get distributed event notifications
- When in debug mode, file logging is enabled, allowing portal log streaming to work

However, in SeaBreeze, there is no shared file system across the containers. Since the portal is connected to only 1 of the N container instances, this means only one of them will be in debug mode. The lack of distributed file change notifications means the others don’t switch to debug mode. Therefore you get spotty log behavior – only the instance in debug mode is logging to files.

To address this, we could have a storage based implementation of our IDebugStateProvider service. The storage based implementation would monitor a blob, and would use periodic polling (with exponential backoff) to detect changes. E.g.:

- Initially no containers running for the app
- User opens app in portal and runs an http function
- The host instance that served the request, “touches” the debug_sentinel blob
- Assuming 2 or more host instances were started by the request(s), the other instances are polling the blob and will see the change and also enter debug mode.

We have discussed having the invoke API the portal uses to run a function return the logs. However that wouldn’t handle queue triggered functions, etc. so really wouldn’t be a comprehensive solution.

Start by locating the IDebugStateProvider interface and the existing debug_sentinel file-based implementation. Trace how the host enters and times out of debug mode, then determine the Blob storage integration points for touching and polling the sentinel with exponential backoff. Done means multiple container instances detect the change and enter debug mode without relying on a shared file system.

Written by the indexing model from the issue text.