[EKS] [request]: Better support for removing instance metadata endpoint access

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**

Removing access to the instance metadata endpoint is documented as good security posture in your documentation: https://docs.aws.amazon.com/eks/latest/userguide/restrict-ec2-credential-access.html. However, there are a couple of improvements that could be made here:

* (less important): This feels like it could be a checkbox somewhere in EKS. I'm not sure if I should have to use a custom launch template with userdata to achieve this.
* (more important): Performing the actions in the documentation linked above (with a custom launch template and custom userdata) stops the Amazon Cloudwatch agent from working. Logs below:

```
2020/09/02 06:07:45 I! 2020/09/02 06:07:42 E! ec2metadata is not available
2020/09/02 06:07:42 I! attempt to access ECS task metadata to determine whether I'm running in ECS.
2020/09/02 06:07:43 W! retry [0/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get http://169.254.170.2/v2/metadata: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020/09/02 06:07:44 W! retry [1/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get http://169.254.170.2/v2/metadata: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020/09/02 06:07:45 W! retry [2/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get http://169.254.170.2/v2/metadata: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020/09/02 06:07:45 I! access ECS task metadata fail with response unable to get response from http://169.254.170.2/v2/metadata, error: Get http://169.254.170.2/v2/metadata: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), assuming I'm not running in ECS.
I! Detected the instance is OnPrem
2020/09/02 06:07:45 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
/opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
2020/09/02 06:07:45 Reading json config file path: /etc/cwagentconfig/..2020_09_02_04_57_01.343707504/cwagentconfig.json ...
2020/09/02 06:07:45 Find symbolic link /etc/cwagentconfig/..data
2020/09/02 06:07:45 Find symbolic link /etc/cwagentconfig/cwagentconfig.json
2020/09/02 06:07:45 Reading json config file path: /etc/cwagentconfig/cwagentconfig.json ...
Valid Json input schema.
Got Home directory: /root
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded

2020/09/02 06:07:45 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2020/09/02 06:07:45 I! AmazonCloudWatchAgent Version 1.245315.0.
2020-09-02T06:07:45Z I! will use file based credentials provider
2020-09-02T06:07:45Z I! Starting AmazonCloudWatchAgent (version 1.245315.0)
2020-09-02T06:07:45Z I! Loaded outputs: cloudwatchlogs
2020-09-02T06:07:45Z I! Loaded inputs: cadvisor k8sapiserver
2020-09-02T06:07:45Z I! Tags enabled:
2020-09-02T06:07:45Z I! Agent Config: Interval:1m0s, Quiet:false, Hostname:"ip-172-21-189-66.ap-southeast-2.compute.internal", Flush Interval:1s
2020-09-02T06:07:45Z I! k8sapiserver Switch New Leader: ip-172-21-188-238.ap-southeast-2.compute.internal
2020-09-02T06:08:06Z E! ec2tagger: Unable to retrieve InstanceId. This plugin must only be used on an EC2 instance
```

I suspect the Cloudwatch Agent needs access to the instance metadata service. However it feels like it shouldn't, and should be able to collect the information it needs via other IRSA permissions, and possibly by looking at the labels on the node (if all it really needs is the instance id)

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
Trying to run the Cloudwatch Agent using IRSA, in a cluster with a strong security posture that has had the instance metadata endpoint disabled for use by pods.

**Are you currently working around this issue?**
You would need to use something else for network policy enforcement, such as Calico. However this comes with extra overhead of managing another cluster service, and deficiencies with the deployment of Calico resources (having to use calicoctl instead of kubectl for some resources).

Start with the linked EKS metadata-access documentation and the reported CloudWatch Agent log; compare the custom launch-template and user-data workaround with the requested EKS control. Define the supported behavior needed for disabling pod access while keeping the agent functional, then verify it against the reported failures.

Written by the indexing model from the issue text.