CloudWatch Agent Fails on EKS when IMDS is Restricted According to Best Practices
- Dominant language
- Mustache
- Stars
- 1.3k
- Forks
- 1.1k
- Avg merge
- 22m
- Merged PRs (30d)
- 5
Description
When deploying the aws-cloudwatch-metric chart version 0.0.4 with image.tag 1.247347.6b250880 and IRSA mapped to arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy I get the following error in the DaemonSet logs. IMDS is disabled per best practices as I'm using RBAC with IRSA. I'm not sure if this is an actual issue or can be safely ignored.
Logs:
2021/05/04 16:39:28 I! 2021/05/04 16:39:25 E! ec2metadata is not available
2021/05/04 16:39:25 I! attempt to access ECS task metadata to determine whether I'm running in ECS.
2021/05/04 16:39:26 W! retry [0/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/05/04 16:39:27 W! retry [1/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/05/04 16:39:28 W! retry [2/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/05/04 16:39:28 I! access ECS task metadata fail with response unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers), assuming I'm not running in ECS.
I! Detected the instance is OnPrem
2021/05/04 16:39:28 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
/opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
2021/05/04 16:39:28 Reading json config file path: /etc/cwagentconfig/..2021_05_04_16_39_17.050069089/cwagentconfig.json ...
2021/05/04 16:39:28 Find symbolic link /etc/cwagentconfig/..data
2021/05/04 16:39:28 Find symbolic link /etc/cwagentconfig/cwagentconfig.json
2021/05/04 16:39:28 Reading json config file path: /etc/cwagentconfig/cwagentconfig.json ...
Valid Json input schema.
Got Home directory: /root
Got Home directory: /root
I! Set home dir Linux: /root
I! SDKRegionWithCredsMap region: us-west-2
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded
2021/05/04 16:39:28 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2021-05-04T16:39:28Z I! Starting AmazonCloudWatchAgent 1.247347.6
2021-05-04T16:39:28Z I! Loaded inputs: cadvisor k8sapiserver
2021-05-04T16:39:28Z I! Loaded aggregators:
2021-05-04T16:39:28Z I! Loaded processors: ec2tagger k8sdecorator
2021-05-04T16:39:28Z I! Loaded outputs: cloudwatchlogs
2021-05-04T16:39:28Z I! Tags enabled:
2021-05-04T16:39:28Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-10-0-6-146.us-west-2.compute.internal", Flush Interval:1s
2021-05-04T16:39:28Z I! [logagent] starting
2021-05-04T16:39:28Z I! [logagent] found plugin cloudwatchlogs is a log backend
Contributor guide
Research direction
Start with the aws-cloudwatch-metric chart version 0.0.4, its DaemonSet, and the CloudWatch Agent image 1.247347.6b250880. Reproduce the deployment with IMDS restricted and IRSA enabled, then determine whether the ec2metadata and ECS metadata errors affect metrics or are harmless. Done means the behavior is clarified and any required chart or configuration change is identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, helm, kubernetes
- Domain
- cloud, devops, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100