aws / aws/amazon-cloudwatch-agent
[Windows] CW Agent silent metric loss after Windows Update - no error logged, no recovery, requires reboot of instance
- Dominant language
- Go
- Stars
- 550
- Forks
- 271
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 13
Description
**Describe the bug**
After a Windows Update reconfigures installed MSI products (triggering `Windows Modules Installer`), PDH performance counter handles become invalid. The `win_perf_counters` plugin then fails on every scrape cycle with `pdhErr=258` but:
1. The error is only logged at `D!` (Debug) level via `log.Printf("D! metric init has error: %v", err)`
2. The `Gather()` function returns `nil` (no error) to the caller
3. The OTel scrape controller in `receiver/adapter/receiver.go` treats this as a successful scrape with zero metrics
4. No metrics are sent to CloudWatch, but the agent continues running with no visible indication of failure
This results in silent metric loss in production. The only way to detect the problem is through external alarms on missing datapoints. The only recovery is to restart the agent process.
**Steps to reproduce**
1. Run CloudWatch Agent on Windows with `win_perf_counters` collecting `LogicalDisk` metrics
2. Allow Windows Update to run (specifically `Windows Modules Installer` / TrustedInstaller MSI reconfiguration)
3. After the update, the agent continues running but stops sending metrics
**What did you expect to see?**
- The agent should either recover by re-initializing PDH counter handles, or
- Log an `E!` (Error) level message indicating metrics collection has failed, or
- Return an error from `Gather()` so the OTel pipeline can report the failure
**What did you see instead?**
Agent process running, log file growing. Zero metrics reaching CloudWatch. CloudWatch alarm triggered after 70 minutes of missing data. Agent does not self-recover.
Service stop succeeded (SCM reported stopped), but agent processes remained running.
Service restart created a third process and all three coexisted, none delivering metrics. `taskkill /F` failed with Access Denied. Only an instance reboot resolved the situation, causing production downtime.
Agent log after PDH counter invalidation (debug level enabled):
```
# First failed scrape:
2026-07-11T18:50:42Z D! {"caller":"adapter/receiver.go:61","msg":"Begin scraping metrics with adapter","receiver":"win_perf_counters"}
2026-07-11T18:50:43Z D! metric init has error: (pdhErr=258) The system cannot find message text for message number 0x%1 in the message file for %2.
# Same error repeating indefinitely without escalation or recovery, only resolvable via instance reboot:
2026-07-14T11:41:42Z D! {"caller":"adapter/receiver.go:61","msg":"Begin scraping metrics with adapter","receiver":"win_perf_counters"}
2026-07-14T11:41:43Z D! metric init has error: (pdhErr=258) The system cannot find message text for message number 0x%1 in the message file for %2.
# Note: At default log level (info/warn), these lines would be completely invisible.
# No E! or W! is ever emitted. Gather() returns nil. Agent reports healthy.
```
Note: The logged value `pdhErr=258` may represent a PDH status code that `FormatMessage()` cannot resolve (the error text "The system cannot find message text..." is the FormatMessage failure itself, not the PDH error description). The underlying PDH status is unclear from the log alone.
**What version did you use?**
Version: `CWAgent/1.300069.0b1529 (go1.26.3; windows; amd64)`
**What config did you use?**
Agent configured with `win_perf_counters` collecting `LogicalDisk % Free Space` metrics with dimensions `[instance = C:]`, `[InstanceId]`, `[objectname = LogicalDisk]`. Namespace: custom namespace. Collection interval: 60s.
**Environment**
OS: Windows Server 2016 (EC2, eu-central-1)
Agent running as: Windows Service (AmazonCloudWatchAgent)
**Additional context**
We traced the behavior to the following code pattern in `plugins/inputs/win_perf_counters/win_perf_counters.go` (`Gather()` method):
Note: Code references are based on the current `main` branch. The CWA fork may differ slightly from Telegraf upstream.
```go
if !metric.initialized {
if err := metric.init(); err != nil {
log.Printf("D! metric init has error: %v", err)
continue // skips this metric silently
}
}
// ...
return nil // always returns nil, never an error
```
The `item.init()` method calls `PdhOpenQuery` and `PdhAddEnglishCounter`. After Windows Update invalidates the counter registry, these calls fail. The `initialized` field remains `false`, so `init()` is retried every cycle, but the error is never escalated.
In `receiver/adapter/receiver.go`, `scrape()` checks the return value of `Gather()`:
```go
if err := r.input.Input.Gather(r.accumulator); err != nil {
// never reached because Gather() always returns nil
}
return r.accumulator.GetOtelMetrics(), nil // empty metrics, no error
```
Possible approaches (just ideas, not prescriptive):
- Escalate to `W!` or `E!` after N consecutive `init()` failures
- Return an error from `Gather()` when all metrics fail to initialize
- Attempt a full PDH handle re-initialization (close + reopen) periodically
The trigger in our case correlates with Windows Event ID 1035 (MsiInstaller reconfiguration of ~60 products including VC++ runtimes and the CloudWatch Agent MSI itself), caused by routine Windows Update activity (`Windows Modules Installer` / TrustedInstaller). The first missing CloudWatch datapoint aligns with this activity.
Related: #223 (Windows service two-process model)
**Limitations of this report**
- We did not run `lodctr /R`, `winmgmt /resyncperf`, or `typeperf "\LogicalDisk(*)\% Free Space"` before rebooting, so we cannot confirm whether PDH was broken system-wide or only within the agent process.
- No `Microsoft-Windows-PerfCtrs` events (ID 1008/1023) were found in the event log, but we did not specifically filter for this source at the time.
Contributor guide
Research direction
Start in plugins/inputs/win_perf_counters/win_perf_counters.go at Gather() and item.init(), then trace the error handling in receiver/adapter/receiver.go. Reproduce or inspect the repeated pdhErr=258 initialization failure and determine how failed collection should be surfaced or recovered; done means the agent no longer reports an empty scrape as successful and the behavior is covered by appropriate tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100