aws / aws/amazon-cloudwatch-agent

```awsentity/service/telegraf``` processor auto-injected into statsd pipeline silently drops all custom metrics in agent versions 1.300031+

Open
#2,180 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
550
Forks
271
Avg merge
1d 21h
Merged PRs (30d)
13

Description

**Describe the bug**
When configuring the CloudWatch agent with a statsd-only custom metrics configuration on EC2,
all metrics are silently dropped and never published to CloudWatch. The agent receives statsd
packets, scrapes them on schedule, but never calls `PutMetricData`. No error is logged even
in debug mode.

The root cause appears to be that the config translator automatically injects the
`awsentity/service/telegraf` processor into the statsd pipeline regardless of whether
Application Signals is configured. This processor has `scrape_datapoint_attribute: true`,
which requires entity metadata attributes on metric datapoints that standard statsd metrics
never carry, causing them to be silently dropped before reaching the CloudWatch exporter.

**Steps to reproduce**
1. Launch an EC2 instance (Ubuntu 24.04, amd64)
2. Install CloudWatch agent version 1.300069 via SSM `AWS-ConfigureAWSPackage`
3. Create the following minimal configuration and load it with `fetch-config`:

```json
{
"agent": {
"metrics_collection_interval": 60
},
"metrics": {
"namespace": "MyNamespace",
"metrics_collected": {
"statsd": {
"service_address": ":8125",
"metrics_collection_interval": 60,
"metrics_aggregation_interval": 60
}
}
}
}
```

4. Send a statsd gauge: `echo -n "TestMetric:1|g" > /dev/udp/127.0.0.1/8125`
5. Wait 90 seconds
6. Check CloudWatch — no metrics appear
7. Check agent log — no publish attempt, no error

**What did you expect to see?**
`TestMetric` published to `MyNamespace` in CloudWatch every 60 seconds.

**What did you see instead?**
The agent receives and scrapes statsd packets (confirmed via tcpdump on lo:8125 and the
debug log line `Begin scraping metrics with adapter receiver=statsd`) but never publishes.
No error is logged even with `debug: true`.

The generated YAML pipeline shows the `awsentity` processor being injected automatically
despite Application Signals not being configured anywhere in the JSON config:

```yaml
processors:
awsentity/service/telegraf:
entity_type: Service
platform: ec2
scrape_datapoint_attribute: true

pipelines:
metrics/hostCustomMetrics:
exporters:
- awscloudwatch
processors:
- awsentity/service/telegraf
receivers:
- telegraf_statsd
```

The following warning appears in debug logs:
```
Failed to get service name from instance tags. This is likely because instance tag is
not enabled for IMDS but will not affect agent functionality.
```
Despite the claim that this "will not affect agent functionality", all statsd metrics are
silently dropped. Enabling instance tags in IMDS via:

```bash
aws ec2 modify-instance-metadata-options \
--instance-id \
--instance-metadata-tags enabled \
--region us-east-1
```

...and restarting the agent does not resolve the issue. Metrics continue to be dropped.

**What version did you use?**
Version: `v1.300069.0`

**What config did you use?**

```json
{
"agent": {
"metrics_collection_interval": 60
},
"metrics": {
"namespace": "MyNamespace",
"metrics_collected": {
"statsd": {
"service_address": ":8125",
"metrics_collection_interval": 60,
"metrics_aggregation_interval": 60
}
}
}
}
```

**Environment**
OS: Ubuntu 24.04.4 LTS (Noble Numbat), amd64
EC2 instance type: t3.small
Region: us-east-1
Installation method: SSM `AWS-ConfigureAWSPackage`
IAM permissions: `cloudwatch:PutMetricData` confirmed working independently via awscli v2

**Additional context**
- The config wizard (`amazon-cloudwatch-agent-config-wizard`) generates the same broken
pipeline with `awsentity` injected — there is no wizard option to opt out
- No error or warning is logged when metrics are dropped, making this extremely difficult
to diagnose
- The workaround is to bypass the agent entirely and call
`aws cloudwatch put-metric-data` directly from a script
- This is a regression — statsd custom metrics worked correctly in agent versions prior
to the introduction of Application Signals pipeline injection (pre-1.300031)

Contributor guide

Open the contributing guide

Research direction

Start with the config translator and the generated YAML pipeline for the statsd-only configuration described here. Verify how awsentity/service/telegraf is selected, then run the reproduced statsd setup and confirm that metrics reach PutMetricData without Application Signals configuration. Done means the statsd metric is published and coverage protects against unconditional processor injection.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.