aws / aws/amazon-cloudwatch-agent

Windows: agent hangs at system shutdown when running as a Windows service (SIGTERM path not terminated)

Open
#2,214 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
550
Forks
271
Avg merge
1d 21h
Merged PRs (30d)
13

Description

**Describe the bug**

When the CloudWatch Agent runs as a Windows service on hosts where the collector is spawned as a console-attached child of `start-amazon-cloudwatch-agent.exe`, an OS shutdown can hang for the duration of `WaitToKillServiceTimeout` (up to several minutes when the value is raised from the default). The system event log records `Event ID 6008` (unexpected shutdown) and `Kernel-Power 41`. On EC2 the hang exceeds the platform's ~4-minute hard-stop and the instance is force-powered-off.

Root cause (short form): Windows shutdown delivers two independent notifications:
(1) csrss / `CsrShutdownProcesses` sends `CTRL_SHUTDOWN_EVENT` to console-attached processes and waits for them to exit (the Go runtime maps this to `SIGTERM`), then (2) the SCM's `ScmAutoShutdownServices` delivers `SERVICE_CONTROL_SHUTDOWN` to registered services.
The collector is spawned as a **child** of `start-amazon-cloudwatch-agent.exe`. csrss delivers `CTRL_SHUTDOWN` to the child, but the SCM routes `SERVICE_CONTROL_SHUTDOWN` to the SCM-registered launcher process, not the child. Inside `kardianos/service` v1.2.1's non-interactive `windowsService.Run()`, the code waits solely on the SCM control channel and does not watch `SIGTERM`.

Result:
- otelcol receives `SIGTERM` via `signal.Notify`, does its graceful shutdown, and logs `Shutdown complete.`
- `cmd.Execute()` returns, `runAgent()` returns, `reloadLoop` returns, `(*program).run()` returns.
- [The outer `svc.Run()` in kardianos](https://github.com/kardianos/service/blob/v1.2.1/service_windows.go#L191) is still blocked waiting for `SERVICE_CONTROL_SHUTDOWN` that never arrives; `main` never returns; the Go runtime's `ctrlHandler` is parked in [`block()`](https://go.googlesource.com/go/%2B/master/src/runtime/os_windows.go#1047); the process does not exit.
- csrss is still waiting for the process to exit -> the whole shutdown deadlocks until the platform's hard-timeout expires.

`sc stop AmazonCloudWatchAgent` (SCM STOP via `ControlService`) is unaffected because it goes through the service control pipe and `prg.Stop` handles it normally.

**Steps to reproduce**

1. Install the CloudWatch Agent MSI on Windows Server. Reproduced against `1.300067.0b1404` and confirmed present on the current `main` (kardianos/service v1.2.1 is still pinned).
2. Configure any minimal JSON that starts the agent as a Windows service (attached below).
3. Set `HKLM\SYSTEM\CurrentControlSet\Control\WaitToKillServiceTimeout` to a value large enough to exceed the platform hard-stop, e.g. `300000` (5 minutes). The default `20000` masks the deadlock by force-killing the process; the collector is still hung, but Windows terminates it before the 4-minute EC2 hard-stop hits.
4. Reboot to apply `WaitToKillServiceTimeout`, then trigger an OS shutdown (`shutdown /s /t 0` or, on EC2, `StopInstances`).
5. Observe:
- Shutdown duration approximately equals `WaitToKillServiceTimeout` (or the platform hard-stop,
whichever is smaller).
- System event log: `1074` (shutdown initiated), no `6006`, `41` (Kernel-Power), `6008`
(previous system shutdown was unexpected).
- `amazon-cloudwatch-agent.log` ends with `Shutdown complete.` while the process is still alive.

**What did you expect to see?**

An OS shutdown that completes cleanly (System event `6006`, no `6008`, no `41`) whether the agent is stopped via SCM STOP or via `CTRL_SHUTDOWN`.

**What did you see instead?**

OS shutdown hangs ~290 s (with `WaitToKillServiceTimeout=300000`) and results in `6008` + `Kernel-Power 41`. Agent log stops updating after `Shutdown complete.` while the process remains alive until Windows force-kills it.

**What version did you use?**

Reproduced against `1.300067.0b1404`. The relevant code path exists on `main` (`cmd/amazon-cloudwatch-agent/amazon-cloudwatch-agent.go` + `github.com/kardianos/service v1.2.1`).

**What config did you use?**

Minimal Windows Event Log collector config (identical failure with any config that starts the agent as a service):

```json
{
"logs": {
"logs_collected": {
"windows_events": {
"collect_list": [
{
"event_name": "System",
"event_levels": ["INFORMATION","WARNING","ERROR","CRITICAL"],
"log_group_name": "repro/System",
"log_stream_name": "{instance_id}"
}
]
}
}
}
}
```

**Environment**

OS: Windows Server 2022 Datacenter (JP; EN reproduces identically), on EC2 m5.xlarge in ap-northeast-1. Reproduced on stock Windows Server 2025 as well when `WaitToKillServiceTimeout` is set high enough to expose the same deadlock; the OS version is not the differentiator.

Contributor guide

Open the contributing guide

Research direction

Start in cmd/amazon-cloudwatch-agent/amazon-cloudwatch-agent.go and trace the service entry point through kardianos/service v1.2.1, especially service_windows.go:191. Reproduce with the Windows service and shutdown procedure described, then verify that both SCM STOP and CTRL_SHUTDOWN allow the process to exit cleanly without exceeding WaitToKillServiceTimeout or producing unexpected-shutdown events.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.