aws / aws/amazon-cloudwatch-agent

amazon-cloudwatch-agent doesnt respect SIGTERM to flush metrics

Open
#961 10 comments 6 reactions 0 assignees View on GitHub
status/backlog under-investigation
Dominant language
Go
Stars
550
Forks
271
Avg merge
1d 21h
Merged PRs (30d)
13

Description

**Describe the bug**

I am trying to run the aws-cloudwatch-agent as part of a lambda extension, so that metrics sent by lambda executions are batched.
Basically, I have a go program that runs this command under the hood with some static configuration files that are pasted below
`amazon-cloudwatch-agent -config config.toml -otelconfig config.yaml`
So far, the bootup works. However, the problem is that the agent doesnt seem to flush the metrics when it receives a SIGTERM signal. It just seems to close all extensions and shutdown without flushing any metrics to cloudwatch. This is really problematic for using inside a lambda extension because until a lambda environment shutdown if the flush interval is not reached, metrics are lost completely

Apparently, the telegraf agent understands a SIGTERM or SIGUSR1 and flushes its cache before shutting down.

[This issue](https://github.com/aws/amazon-cloudwatch-agent/issues/170) already goes in this direction.

**What did you expect to see?**
The metrics should be flushed during shutdown

**What did you see instead?**
The metrics are lost

**What config did you use?**
```
[agent]
collection_jitter = "0s"
debug = false
flush_interval = "1s"
flush_jitter = "0s"
hostname = ""
interval = "60s"
metric_batch_size = 1000
metric_buffer_limit = 10000
omit_hostname = true
precision = ""
quiet = false
round_interval = false

[inputs]

[[inputs.statsd]]
interval = "10s"
parse_data_dog_tags = true
service_address = ":8125"
[inputs.statsd.tags]
"aws:AggregationInterval" = "10s"

[outputs]

[[outputs.cloudwatch]]
```

config.yaml
```
connectors: {}
exporters:
awscloudwatch:
force_flush_interval: 10s
max_datums_per_call: 1000
max_values_per_datum: 150
namespace: somewhere
region: eu-central-1
resource_to_telemetry_conversion:
enabled: true
extensions: {}
processors: {}
receivers:
telegraf_statsd:
collection_interval: 10s
initial_delay: 1s
timeout: 0s
service:
extensions: []
pipelines:
metrics/host:
exporters:
- awscloudwatch
processors: []
receivers:
- telegraf_statsd
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the shutdown behavior with the supplied config.toml and config.yaml, then trace the agent's SIGTERM handling and shutdown sequence. Confirm whether buffered metrics are flushed before termination; done means metrics emitted before SIGTERM reach CloudWatch instead of being lost.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.