Azure / Azure/azure-functions-host
Host health monitor discards exceeded counter names, making "Host is unhealthy" difficult to diagnose
- Dominant language
- C#
- Stars
- 2k
- Forks
- 482
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 38
Description
### Problem
When the periodic host health check trips, the host logs `Host is unhealthy. Initiating a restart.` but never says *which* counter exceeded.
[`IsHostHealthy`](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs#L848-L867) collects the names into `exceededCounters`, formats them, and then drops them at [line 863](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs#L863) — the `Host thresholds exceeded: [...]` message only exists in the `throwWhenUnhealthy` branch. [`OnHostHealthCheckTimer`](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs#L832-L846) is the only caller using the default `false`, so on the periodic path that message is unreachable. No log level or configuration setting can surface it.
### Why it matters
The counters ([`Connections`, `Sections`, `Threads`, etc.](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script/Scale/HostPerformanceManager.cs#L132-L146)) come from the `WEBSITE_COUNTERS_APP` environment variable and are not published to Azure Monitor or Application Insights — if the host does not log them, they are unavailable from every direction.
Recent customer case: a Windows Consumption app restarting every 7–11 minutes. Weeks were spent correctly eliminating memory, threads, timeouts, SNAT and sampling, plus raising the log level as guidance suggests. The cause was one counter sitting slightly over the [80% threshold](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script/Config/HostHealthMonitorOptions.cs#L12). One log line would have made it a five-minute diagnosis.
### Proposed fix
Add a `HostThresholdsExceeded` message to [`ScriptHostServiceLoggerExtension`](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script.WebHost/Diagnostics/Extensions/ScriptHostServiceLoggerExtension.cs#L51-L55) at `LogLevel.Warning` (next free EventId is 532), and call it before `return false`:
```csharp
_logger.HostThresholdsExceeded(formattedCounters);
```
Checks run every 10s; if log volume is a concern, log only when the counter set *changes*.
### Also worth fixing
- [`HttpThrottleMiddleware`](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script.WebHost/Middleware/HttpThrottleMiddleware.cs#L54) logs different wording for the same condition, so a search for one misses the other.
- Document the **80%** threshold — customers compare against the published limit with no way to know the effective ceiling is 80% of it.
### Environment
Observed on host `4.1053.200`, Windows Consumption. The code path is long-standing and not specific to that build; it applies to any SKU where `ShouldMonitorHostHealth` is true.
---
Written by Azure SRE Agent following a customer incident investigation.
Contributor guide
Research direction
Start in WebJobsScriptHostService.cs at IsHostHealthy and OnHostHealthCheckTimer, then inspect ScriptHostServiceLoggerExtension.cs for the existing logger messages and event IDs. Confirm the periodic health-check path records the exceeded counter names at warning level before returning false, using the proposed HostThresholdsExceeded message as the completion criterion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100