aws-samples / aws-samples/amazon-cloudwatch-container-insights
AWS cloud agent deployment failure on ROSA
- Dominant language
- Shell
- Stars
- 178
- Forks
- 117
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 3
Description
Hi team, I have a Rosa cluster deployed in region `us-east-2` by following https://console.redhat.com/openshift/create/rosa/welcome
I am trying to setup the cloudwatch agent to collect the metrics by following.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-EKS-quickstart.html
Before installation
We attached CloudWatchAgentServerPolicy to each EC2 worker nodes.
But during daemonset deployment we got error:
```
[telegraf] Error running agent: could not initialize processor ec2tagger: ec2tagger: Unable to retrieve InstanceId. This plugin must only be used on an EC2 instance`
```
detail log:
```
`2021/11/19 10:04:19 I! 2021/11/19 10:04:16 E! ec2metadata is not available
2021/11/19 10:04:16 I! attempt to access ECS task metadata to determine whether I'm running in ECS.
2021/11/19 10:04:17 W! retry [0/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/11/19 10:04:18 W! retry [1/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/11/19 10:04:19 W! retry [2/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/11/19 10:04:19 I! access ECS task metadata fail with response unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers), assuming I'm not running in ECS.
I! Detected the instance is OnPrem
2021/11/19 10:04:19 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
/opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
2021/11/19 10:04:19 Reading json config file path: /etc/cwagentconfig/..2021_11_19_09_40_24.899664046/cwagentconfig.json ...
2021/11/19 10:04:19 Find symbolic link /etc/cwagentconfig/..data
2021/11/19 10:04:19 Find symbolic link /etc/cwagentconfig/cwagentconfig.json
2021/11/19 10:04:19 Reading json config file path: /etc/cwagentconfig/cwagentconfig.json ...
Valid Json input schema.
Got Home directory: /root
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded
2021/11/19 10:04:19 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2021-11-19T10:04:19Z I! Starting AmazonCloudWatchAgent 1.247348.0
2021-11-19T10:04:19Z I! Loaded inputs: cadvisor k8sapiserver
2021-11-19T10:04:19Z I! Loaded aggregators:
2021-11-19T10:04:19Z I! Loaded processors: ec2tagger k8sdecorator
2021-11-19T10:04:19Z I! Loaded outputs: cloudwatchlogs
2021-11-19T10:04:19Z I! Tags enabled:
2021-11-19T10:04:19Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-10-0-137-118.us-east-2.compute.internal", Flush Interval:1s
2021-11-19T10:04:19Z I! [logagent] starting
2021-11-19T10:04:19Z I! [logagent] found plugin cloudwatchlogs is a log backend
2021-11-19T10:04:28Z E! [processors.ec2tagger] ec2tagger: Unable to retrieve InstanceId. This plugin must only be used on an EC2 instance
2021-11-19T10:04:28Z E! [telegraf] Error running agent: could not initialize processor ec2tagger: ec2tagger: Unable to retrieve InstanceId. This plugin must only be used on an EC2 instance
```
Here is my cluster info:
```
yuri9doo86@loaclhost aws % ..//rosa describe cluster -c cherryrosatest
Name: cherryrosatest
ID: 1ogfk2f7f49ah56deg0neq1qks1cgbs2
External ID: b136164e-766b-4ab0-a71e-7b4415ed4663
OpenShift Version: 4.9.5
Channel Group: stable
DNS: cherryrosatest.3y19.p1.openshiftapps.com
AWS Account: XXXXXX
API URL: https://api.cherryrosatest.3y19.p1.openshiftapps.com:6443
Console URL: https://console-openshift-console.apps.cherryrosatest.3y19.p1.openshiftapps.com
Region: us-east-2
Multi-AZ: false
Nodes:
- Control plane: 3
- Infra: 2
- Compute (Autoscaled): 4-10
Network:
- Service CIDR: 172.30.0.0/16
- Machine CIDR: 10.0.0.0/16
- Pod CIDR: 10.128.0.0/14
- Host Prefix: /23
STS Role ARN: arn:aws:iam::675801125365:role/ManagedOpenShift-Installer-Role
Support Role ARN: arn:aws:iam::675801125365:role/ManagedOpenShift-Support-Role
Instance IAM Roles:
- Control plane: arn:aws:iam::675801125365:role/ManagedOpenShift-ControlPlane-Role
- Worker: arn:aws:iam::675801125365:role/ManagedOpenShift-Worker-Role
Operator IAM Roles:
- arn:aws:iam::675801125365:role/cherryrosatest-s9a5-openshift-machine-api-aws-cloud-credentials
- arn:aws:iam::675801125365:role/cherryrosatest-s9a5-openshift-cloud-credential-operator-cloud-cr
- arn:aws:iam::675801125365:role/cherryrosatest-s9a5-openshift-image-registry-installer-cloud-cre
- arn:aws:iam::675801125365:role/cherryrosatest-s9a5-openshift-ingress-operator-cloud-credentials
- arn:aws:iam::675801125365:role/cherryrosatest-s9a5-openshift-cluster-csi-drivers-ebs-cloud-cred
State: ready
Private: No
Created: Nov 16 2021 05:56:28 UTC
Details Page: https://console.redhat.com/openshift/details/s/20zKtEuEok2WNe2NVsKQwtEF7Fh
OIDC Endpoint URL: https://rh-oidc.s3.us-east-1.amazonaws.com/1ogfk2f7f49ah56deg0neq1qks1cgbs2
```
Contributor guide
Research direction
Start with the AWS Container Insights setup instructions and the repository's Kubernetes YAML templates, then compare the daemonset configuration with the ROSA cluster details and logged metadata failure. Done means determining whether the deployment supports this ROSA environment and identifying the required configuration or documentation change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100