aws / aws/amazon-cloudwatch-agent

Graceful shutdown

Open
#170 3 comments 5 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
550
Forks
271
Avg merge
1d 21h
Merged PRs (30d)
13

Description

How should I configure the CloudWatch Agent to make sure all collected data pushed away when I stop the agent?
I tried many options, but wasn't able to make it work as expected.

We use metrics to collect statistics in our application. Thus, we expect the metrics to be precise enough.
We have several ASG clusters with installed CloudWatch Agent with StatsD plugin on each EC2. There are several modules sending metrics to the agent. We found some metrics lost when ASG scales down. During investigation and some manual tests we found the agent doesn't publish collected data even if the amazon-cloudwatch-agent service stopped gracefully.

Can it be just incorrect configuration or WAD feature?

My configuration:
```
{
"agent": {
"metrics_collection_interval": 60,
"omit_hostname": true,
"logfile": "/var/log/amazon-cloudwatch-agent.log"
},
"metrics": {
"namespace": "MyNameSpace",
"append_dimensions": {
"AutoScalingGroupName": "${aws:AutoScalingGroupName}"
},
"metrics_collected": {
"statsd": {
"metrics_collection_interval": 60,
"metrics_aggregation_interval": 0
},
"mem": {
"metrics_collection_interval": 60,
"metrics_aggregation_interval": 0,
"measurement": [
"used_percent"
]
}
}
}
}
```

For manual test we used:
* start the agent service
* send 1000 data points for a single metric by netcat
* wait about 20 seconds
* stop the agent service

Thanks.

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named in the issue. Reproduce the reported sequence with the supplied configuration: send 1,000 StatsD points, wait about 20 seconds, and gracefully stop the agent; then trace shutdown handling and metric export flushing. Done means collected points are published before the service exits, or the configuration limitation is documented clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.