aws / aws/amazon-cloudwatch-agent
Graceful shutdown
- Dominant language
- Go
- Stars
- 550
- Forks
- 271
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 13
Description
How should I configure the CloudWatch Agent to make sure all collected data pushed away when I stop the agent?
I tried many options, but wasn't able to make it work as expected.
We use metrics to collect statistics in our application. Thus, we expect the metrics to be precise enough.
We have several ASG clusters with installed CloudWatch Agent with StatsD plugin on each EC2. There are several modules sending metrics to the agent. We found some metrics lost when ASG scales down. During investigation and some manual tests we found the agent doesn't publish collected data even if the amazon-cloudwatch-agent service stopped gracefully.
Can it be just incorrect configuration or WAD feature?
My configuration:
```
{
"agent": {
"metrics_collection_interval": 60,
"omit_hostname": true,
"logfile": "/var/log/amazon-cloudwatch-agent.log"
},
"metrics": {
"namespace": "MyNameSpace",
"append_dimensions": {
"AutoScalingGroupName": "${aws:AutoScalingGroupName}"
},
"metrics_collected": {
"statsd": {
"metrics_collection_interval": 60,
"metrics_aggregation_interval": 0
},
"mem": {
"metrics_collection_interval": 60,
"metrics_aggregation_interval": 0,
"measurement": [
"used_percent"
]
}
}
}
}
```
For manual test we used:
* start the agent service
* send 1000 data points for a single metric by netcat
* wait about 20 seconds
* stop the agent service
Thanks.
Contributor guide
Research direction
No source files, tests, or entry points are named in the issue. Reproduce the reported sequence with the supplied configuration: send 1,000 StatsD points, wait about 20 seconds, and gracefully stop the agent; then trace shutdown handling and metric export flushing. Done means collected points are published before the service exits, or the configuration limitation is documented clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100