aws-samples / aws-samples/amazon-cloudwatch-container-insights

AWS CloudWatch Agent failing on OCP 4.6 cluster

Open
#75 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
178
Forks
117
Avg merge
1d 10h
Merged PRs (30d)
3

Description

I am trying to set up aws CloudWatch Agent to Collect ocp 4.6 Cluster Metrics using this link https://docs.amazonaws.cn/en_us/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-metrics.html

but the pods are failing with following errors:
```
[ec2-user@ip-10-0-2-100 ~]$ oc logs cloudwatch-agent-57htk
2021/02/02 21:17:21 I! 2021/02/02 21:17:18 E! ec2metadata is not available
2021/02/02 21:17:18 I! attempt to access ECS task metadata to determine whether I'm running in ECS.
2021/02/02 21:17:19 W! retry [0/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/02/02 21:17:20 W! retry [1/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/02/02 21:17:21 W! retry [2/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/02/02 21:17:21 I! access ECS task metadata fail with response unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers), assuming I'm not running in ECS.
I! Detected the instance is OnPrem
2021/02/02 21:17:21 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
/opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
2021/02/02 21:17:21 Reading json config file path: /etc/cwagentconfig/..2021_02_02_21_14_55.157039078/cwagentconfig.json ...
2021/02/02 21:17:21 unable to scan config dir /etc/cwagentconfig with error: unable to parse json, error: invalid character '\n' in string literal
No json config files found, please provide config, exit now

2021/02/02 21:17:21 I! Return exit error: exit code=99
2021/02/02 21:17:21 I! there is no json configuration when running translator
```

Contributor guide

Open the contributing guide

Research direction

Start with the AWS Container Insights setup link and reproduce the failure from the cloudwatch-agent pod logs. Inspect the /etc/cwagentconfig/.../cwagentconfig.json path and the reported JSON parse error; done means the agent pods start successfully and collect OCP 4.6 cluster metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, devops, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.