Azure / Azure/azure-functions-host

Host health monitor discards exceeded counter names, making "Host is unhealthy" difficult to diagnose

Open Beginner friendly
#12,010 0 comments 0 reactions 0 assignees View on GitHub
area: logging enhancement supportability: error-message-improvement
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 12h
Merged PRs (30d)
38

Description

### Problem

When the periodic host health check trips, the host logs `Host is unhealthy. Initiating a restart.` but never says *which* counter exceeded.

[`IsHostHealthy`](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs#L848-L867) collects the names into `exceededCounters`, formats them, and then drops them at [line 863](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs#L863) — the `Host thresholds exceeded: [...]` message only exists in the `throwWhenUnhealthy` branch. [`OnHostHealthCheckTimer`](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script.WebHost/WebJobsScriptHostService.cs#L832-L846) is the only caller using the default `false`, so on the periodic path that message is unreachable. No log level or configuration setting can surface it.

### Why it matters

The counters ([`Connections`, `Sections`, `Threads`, etc.](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script/Scale/HostPerformanceManager.cs#L132-L146)) come from the `WEBSITE_COUNTERS_APP` environment variable and are not published to Azure Monitor or Application Insights — if the host does not log them, they are unavailable from every direction.

Recent customer case: a Windows Consumption app restarting every 7–11 minutes. Weeks were spent correctly eliminating memory, threads, timeouts, SNAT and sampling, plus raising the log level as guidance suggests. The cause was one counter sitting slightly over the [80% threshold](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script/Config/HostHealthMonitorOptions.cs#L12). One log line would have made it a five-minute diagnosis.

### Proposed fix

Add a `HostThresholdsExceeded` message to [`ScriptHostServiceLoggerExtension`](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script.WebHost/Diagnostics/Extensions/ScriptHostServiceLoggerExtension.cs#L51-L55) at `LogLevel.Warning` (next free EventId is 532), and call it before `return false`:

```csharp
_logger.HostThresholdsExceeded(formattedCounters);
```

Checks run every 10s; if log volume is a concern, log only when the counter set *changes*.

### Also worth fixing

- [`HttpThrottleMiddleware`](https://github.com/Azure/azure-functions-host/blob/ce343352e6974af46670b47c28bab4ce9d93802c/src/WebJobs.Script.WebHost/Middleware/HttpThrottleMiddleware.cs#L54) logs different wording for the same condition, so a search for one misses the other.
- Document the **80%** threshold — customers compare against the published limit with no way to know the effective ceiling is 80% of it.

### Environment

Observed on host `4.1053.200`, Windows Consumption. The code path is long-standing and not specific to that build; it applies to any SKU where `ShouldMonitorHostHealth` is true.

---

Written by Azure SRE Agent following a customer incident investigation.

Contributor guide

Open the contributing guide

Research direction

Start in WebJobsScriptHostService.cs at IsHostHealthy and OnHostHealthCheckTimer, then inspect ScriptHostServiceLoggerExtension.cs for the existing logger messages and event IDs. Confirm the periodic health-check path records the exceeded counter names at warning level before returning false, using the proposed HostThresholdsExceeded message as the completion criterion.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
backend, observability
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.