aws / aws/amazon-cloudwatch-agent

Amazon Cloudwatch Agent Does Not Use Credential File for Inputs: Prometheus Scraper Configuration

Open
#741 6 comments 4 reactions 0 assignees View on GitHub
Stale status/investigate
Dominant language
Go
Stars
550
Forks
271
Avg merge
1d 21h
Merged PRs (30d)
13

Description

# Bug Report

The CloudWatch agent does not appear to utilize the file named `/root/.aws/credentials` as a credential source. The logs below show that the credential file _is_ being used for "outputs" configuration but is not being used by some number of "inputs" configurations in particular `[inputs.prometheus_scraper.ecs_service_discovery.service_name_list_for_tasks]`.

Evidence of credential file being used by "output" - from the logs you can see `shared_credential_file` is set to `shared_credential_file = "/root/.aws/credentials"`.

```
[outputs]

[[outputs.cloudwatchlogs]]
force_flush_interval = "5s"
log_stream_name = "7eccfaad73e7"
profile = "AmazonCloudWatchAgent"
region = "us-west-2"
shared_credential_file = "/root/.aws/credentials"
tagexclude = ["metricPath"]
[outputs.cloudwatchlogs.tagpass]
metricPath = ["logs"]
```

And evidence that the file is _not_ being used by inputs despite being available:

From the logs you can see no evidence that `shared_credential_file = "/root/.aws/credentials"` is used.
```
[inputs]

[[inputs.prometheus_scraper]]
cluster_name = "test"
prometheus_config_path = "/etc/cwagentprometheusconfig/prometheus.yaml"
[inputs.prometheus_scraper.ecs_service_discovery]
sd_cluster_region = "us-west-2"
sd_frequency = "1m"
sd_result_file = "/tmp/cwagent_ecs_auto_sd.yaml"
sd_target_cluster = "ecs-prometheus"

[[inputs.prometheus_scraper.ecs_service_discovery.service_name_list_for_tasks]]
sd_job_name = "prometheus-exporter"
sd_metrics_path = "/metrics"
sd_metrics_ports = "5000"
sd_service_name_pattern = "prometheus-exporter"
[inputs.prometheus_scraper.tags]
log_group_name = "prometheus"
metricPath = "logs"
```

and further:

```
ts=2023-04-22T18:35:02.503Z caller=start.go:325 level=info msg="Add extra relabel_configs and metric_relabel_configs to save job, instance and __name__ before user relabel"
ts=2023-04-22T18:35:02.503Z caller=start.go:342 level=info msg="Completed loading of configuration file" filename=/etc/cwagentprometheusconfig/prometheus.yaml
ts=2023-04-22T18:35:02.503Z caller=start.go:246 level=info msg="finish handling config file"
ts=2023-04-22T18:35:02.503Z caller=start.go:183 level=info msg="start discovery"
2023-04-22T18:35:03Z E! Failed to get credential from session: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: connection refused
2023-04-22T18:35:03Z D! [logagent] open file count, 0
```

# Reproduction

To reproduce the error use the files contained within the "Config" section below.

# Expected Output

The credentials from the credentials file should be used.

# Actual Output

I saw the following errors followed by an application exit:

```
ts=2023-04-22T18:35:02.503Z caller=start.go:325 level=info msg="Add extra relabel_configs and metric_relabel_configs to save job, instance and __name__ before user relabel"
ts=2023-04-22T18:35:02.503Z caller=start.go:342 level=info msg="Completed loading of configuration file" filename=/etc/cwagentprometheusconfig/prometheus.yaml
ts=2023-04-22T18:35:02.503Z caller=start.go:246 level=info msg="finish handling config file"
ts=2023-04-22T18:35:02.503Z caller=start.go:183 level=info msg="start discovery"
2023-04-22T18:35:03Z E! Failed to get credential from session: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: connection refused
2023-04-22T18:35:03Z D! [logagent] open file count, 0
```

# Version

`AmazonCloudWatchAgent 1.247358.0`

# Config

Configuration to reproduce this setup is below:

## docker-compose.yml

```
services:
cloudwatch-agent:
image: public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest
volumes:
- $HOME/.aws/credentials:/root/.aws/credentials:ro
- ./cloudwatch-agent/amazon-cloudwatch-agent-prometheus.json:/etc/cwagentconfig/config.json
- ./cloudwatch-agent/prometheus.yaml:/etc/cwagentprometheusconfig/prometheus.yaml
```

## amazon-cloudwatch-agent-prometheus.json

```
{
"agent":{
"debug": true,
"logfile": "",
"metrics_collection_interval":10
},
"logs":{
"metrics_collected":{
"prometheus":{
"ecs_service_discovery": {
"sd_cluster_region": "us-west-2",
"sd_frequency": "1m",
"sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml",
"sd_target_cluster": "ecs-prometheus",
"service_name_list_for_tasks": [
{
"sd_job_name": "prometheus-exporter",
"sd_metrics_path": "/metrics",
"sd_metrics_ports": "5000",
"sd_service_name_pattern": "prometheus-exporter"
}
]
},
"cluster_name": "test",
"log_group_name":"prometheus",
"prometheus_config_path":"/etc/cwagentprometheusconfig/prometheus.yaml"
}
}
}
}
```

## prometheus.yaml

```
global:
scrape_interval: 1m
scrape_timeout: 10s
scrape_configs:
- job_name: prometheus-exporter
metrics_path: /metrics
scheme: http
file_sd_configs:
- files:
- /tmp/cwagent_ecs_auto_sd.yaml
```

# Environment

I am using the Docker Image `public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest` - which, at this moment, contains CloudWatch agent `1.247358.0`.

# Additional Context

None.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with docker-compose.yml, amazon-cloudwatch-agent-prometheus.json, and prometheus.yaml using the mounted /root/.aws/credentials file. Trace the ECS service discovery configuration and credential loading shown in the logs; done means the Prometheus scraper's ECS discovery uses the configured credential file instead of exiting with NoCredentialProviders.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, go, prometheus
Domain
authentication, backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.