aws-samples / aws-samples/amazon-cloudwatch-container-insights

Workaround for pods not being able to access EC2 IMDS

Open
#56 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
178
Forks
117
Avg merge
1d 10h
Merged PRs (30d)
3

Description

In my EKS cluster I have disabled the Instance Metadata Service (IMDS) v1 and set the IMDS hop count to 1, to prevent pods from accessing the IMDS. This prevents the `cloudwatch-agent` daemonset from starting, since it looks into the EC2 IMDS.

```
2020/11/12 15:56:28 I! 2020/11/12 15:56:25 E! ec2metadata is not available
2020/11/12 15:56:25 I! attempt to access ECS task metadata to determine whether I'm running in ECS.
2020/11/12 15:56:26 W! retry [0/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
```

Is there a config item or workaround to use this agent without allowing pods to access the EC2 IMDS?

Contributor guide

Open the contributing guide

Research direction

Start with the cloudwatch-agent DaemonSet and its configuration, then trace how it discovers EC2 or ECS metadata under IMDSv1-disabled, hop-count-1 settings. Reproduce the startup failure using the logged metadata endpoints and compare available configuration or deployment options; done means the agent starts and collects data without pod access to EC2 IMDS.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.