aws / aws/amazon-cloudwatch-agent
amazon-cloudwatch-agent doesnt respect SIGTERM to flush metrics
- Dominant language
- Go
- Stars
- 550
- Forks
- 271
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 13
Description
**Describe the bug**
I am trying to run the aws-cloudwatch-agent as part of a lambda extension, so that metrics sent by lambda executions are batched.
Basically, I have a go program that runs this command under the hood with some static configuration files that are pasted below
`amazon-cloudwatch-agent -config config.toml -otelconfig config.yaml`
So far, the bootup works. However, the problem is that the agent doesnt seem to flush the metrics when it receives a SIGTERM signal. It just seems to close all extensions and shutdown without flushing any metrics to cloudwatch. This is really problematic for using inside a lambda extension because until a lambda environment shutdown if the flush interval is not reached, metrics are lost completely
Apparently, the telegraf agent understands a SIGTERM or SIGUSR1 and flushes its cache before shutting down.
[This issue](https://github.com/aws/amazon-cloudwatch-agent/issues/170) already goes in this direction.
**What did you expect to see?**
The metrics should be flushed during shutdown
**What did you see instead?**
The metrics are lost
**What config did you use?**
```
[agent]
collection_jitter = "0s"
debug = false
flush_interval = "1s"
flush_jitter = "0s"
hostname = ""
interval = "60s"
metric_batch_size = 1000
metric_buffer_limit = 10000
omit_hostname = true
precision = ""
quiet = false
round_interval = false
[inputs]
[[inputs.statsd]]
interval = "10s"
parse_data_dog_tags = true
service_address = ":8125"
[inputs.statsd.tags]
"aws:AggregationInterval" = "10s"
[outputs]
[[outputs.cloudwatch]]
```
config.yaml
```
connectors: {}
exporters:
awscloudwatch:
force_flush_interval: 10s
max_datums_per_call: 1000
max_values_per_datum: 150
namespace: somewhere
region: eu-central-1
resource_to_telemetry_conversion:
enabled: true
extensions: {}
processors: {}
receivers:
telegraf_statsd:
collection_interval: 10s
initial_delay: 1s
timeout: 0s
service:
extensions: []
pipelines:
metrics/host:
exporters:
- awscloudwatch
processors: []
receivers:
- telegraf_statsd
```
Contributor guide
Research direction
Start by reproducing the shutdown behavior with the supplied config.toml and config.yaml, then trace the agent's SIGTERM handling and shutdown sequence. Confirm whether buffered metrics are flushed before termination; done means metrics emitted before SIGTERM reach CloudWatch instead of being lost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100