aws / aws/amazon-cloudwatch-agent
Windows: agent hangs at system shutdown when running as a Windows service (SIGTERM path not terminated)
- Dominant language
- Go
- Stars
- 550
- Forks
- 271
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 13
Description
**Describe the bug**
When the CloudWatch Agent runs as a Windows service on hosts where the collector is spawned as a console-attached child of `start-amazon-cloudwatch-agent.exe`, an OS shutdown can hang for the duration of `WaitToKillServiceTimeout` (up to several minutes when the value is raised from the default). The system event log records `Event ID 6008` (unexpected shutdown) and `Kernel-Power 41`. On EC2 the hang exceeds the platform's ~4-minute hard-stop and the instance is force-powered-off.
Root cause (short form): Windows shutdown delivers two independent notifications:
(1) csrss / `CsrShutdownProcesses` sends `CTRL_SHUTDOWN_EVENT` to console-attached processes and waits for them to exit (the Go runtime maps this to `SIGTERM`), then (2) the SCM's `ScmAutoShutdownServices` delivers `SERVICE_CONTROL_SHUTDOWN` to registered services.
The collector is spawned as a **child** of `start-amazon-cloudwatch-agent.exe`. csrss delivers `CTRL_SHUTDOWN` to the child, but the SCM routes `SERVICE_CONTROL_SHUTDOWN` to the SCM-registered launcher process, not the child. Inside `kardianos/service` v1.2.1's non-interactive `windowsService.Run()`, the code waits solely on the SCM control channel and does not watch `SIGTERM`.
Result:
- otelcol receives `SIGTERM` via `signal.Notify`, does its graceful shutdown, and logs `Shutdown complete.`
- `cmd.Execute()` returns, `runAgent()` returns, `reloadLoop` returns, `(*program).run()` returns.
- [The outer `svc.Run()` in kardianos](https://github.com/kardianos/service/blob/v1.2.1/service_windows.go#L191) is still blocked waiting for `SERVICE_CONTROL_SHUTDOWN` that never arrives; `main` never returns; the Go runtime's `ctrlHandler` is parked in [`block()`](https://go.googlesource.com/go/%2B/master/src/runtime/os_windows.go#1047); the process does not exit.
- csrss is still waiting for the process to exit -> the whole shutdown deadlocks until the platform's hard-timeout expires.
`sc stop AmazonCloudWatchAgent` (SCM STOP via `ControlService`) is unaffected because it goes through the service control pipe and `prg.Stop` handles it normally.
**Steps to reproduce**
1. Install the CloudWatch Agent MSI on Windows Server. Reproduced against `1.300067.0b1404` and confirmed present on the current `main` (kardianos/service v1.2.1 is still pinned).
2. Configure any minimal JSON that starts the agent as a Windows service (attached below).
3. Set `HKLM\SYSTEM\CurrentControlSet\Control\WaitToKillServiceTimeout` to a value large enough to exceed the platform hard-stop, e.g. `300000` (5 minutes). The default `20000` masks the deadlock by force-killing the process; the collector is still hung, but Windows terminates it before the 4-minute EC2 hard-stop hits.
4. Reboot to apply `WaitToKillServiceTimeout`, then trigger an OS shutdown (`shutdown /s /t 0` or, on EC2, `StopInstances`).
5. Observe:
- Shutdown duration approximately equals `WaitToKillServiceTimeout` (or the platform hard-stop,
whichever is smaller).
- System event log: `1074` (shutdown initiated), no `6006`, `41` (Kernel-Power), `6008`
(previous system shutdown was unexpected).
- `amazon-cloudwatch-agent.log` ends with `Shutdown complete.` while the process is still alive.
**What did you expect to see?**
An OS shutdown that completes cleanly (System event `6006`, no `6008`, no `41`) whether the agent is stopped via SCM STOP or via `CTRL_SHUTDOWN`.
**What did you see instead?**
OS shutdown hangs ~290 s (with `WaitToKillServiceTimeout=300000`) and results in `6008` + `Kernel-Power 41`. Agent log stops updating after `Shutdown complete.` while the process remains alive until Windows force-kills it.
**What version did you use?**
Reproduced against `1.300067.0b1404`. The relevant code path exists on `main` (`cmd/amazon-cloudwatch-agent/amazon-cloudwatch-agent.go` + `github.com/kardianos/service v1.2.1`).
**What config did you use?**
Minimal Windows Event Log collector config (identical failure with any config that starts the agent as a service):
```json
{
"logs": {
"logs_collected": {
"windows_events": {
"collect_list": [
{
"event_name": "System",
"event_levels": ["INFORMATION","WARNING","ERROR","CRITICAL"],
"log_group_name": "repro/System",
"log_stream_name": "{instance_id}"
}
]
}
}
}
}
```
**Environment**
OS: Windows Server 2022 Datacenter (JP; EN reproduces identically), on EC2 m5.xlarge in ap-northeast-1. Reproduced on stock Windows Server 2025 as well when `WaitToKillServiceTimeout` is set high enough to expose the same deadlock; the OS version is not the differentiator.
Contributor guide
Research direction
Start in cmd/amazon-cloudwatch-agent/amazon-cloudwatch-agent.go and trace the service entry point through kardianos/service v1.2.1, especially service_windows.go:191. Reproduce with the Windows service and shutdown procedure described, then verify that both SCM STOP and CTRL_SHUTDOWN allow the process to exit cleanly without exceeding WaitToKillServiceTimeout or producing unexpected-shutdown events.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100