[EKS] [request]: Simplify EKS managed worker bootstrap args

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**

EKS managed worker bootstrap args without requiring a launch template and explicit AMI (e.g., allow bootstrap args as part of the Node Group API).

**Which service(s) is this request for?**

EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**

I am running a lot of pods on m5.large and t3.medium instances with low CPU and memory utilization, but the default pod limits are 29 and 17 respectively, which would make my instances severely underutilized. To get around this issue, I am using Calico IPAM Instead of the default VPC CNI (I am aware of the recent enhancements to the AWS VPC CNI, but Calico for IPAM is still a simpler solution in my case). However, the [default pod limits](https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt) are still in effect unless I pass in `--use-max-pods false` as an argument to bootstrap.sh in the [EKS AMI](https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh). Managed workers now supports custom launch templates, and in theory, bootstrap args can be passed in via user data. However, EKS managed workers has logic where the service role duplicates my launch template, and merges the default EKS user data with my user data, and there is no simple way to get this mechanism to merge bootstrap args with the default EKS user data that invokes bootstrap.sh. After creating an AWS support ticket, I was informed that I would need to specify the AMI explicitly to disable this merge logic, and then create the entire user data script that invokes bootstrap.sh and passes in my desired bootstrap args, along with the cluster CA, kubelet args, etc., which ends up not being such a simple solution. So, bootstrap args with managed workers is fairly complex to accomplish, even though it's technically possible. In addition, since the AMI has to be explicitly specified, the ability to upgrade to the lastest AMI via the AWS Console is lost.

**Are you currently working around this issue?**

Rather than explicitly specifying the AMI and building my own invocation script to bootstrap.sh in my launch template user data to pass in `--use-max-pods false`, I created a simple user data script that updates [eni-max-pods.txt](https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt) so that any instance sizes with a pod count that is less than the default 110 pods (e.g., 29 for m5.large and 17 for t3.medium) will get updated to be the default 110 pods instead:

```
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
cat /etc/eks/eni-max-pods.txt | awk '($2+0 > 0 && $2+0 <= 110) {print $1 OFS 110;next} 1' > /tmp/eni-max-pods.txt
mv /tmp/eni-max-pods.txt /etc/eks/eni-max-pods.txt

--//--
```

When I place the script above in the user data for my managed worker launch template, the service role copies my launch template, and puts my user data script first, and then the default EKS user data script that invokes bootstrap.sh second. As a result, my managed workers end up with a max pod limit of 110.

Even if AWS doesn't end up simplifying the process of passing bootstrap args so a launch template and explicit AMI are not required, hopefully my workaround above will at least help others who want to use CNIs with alternate IPAM like Calico, Flannel, Weave Net, etc. with managed workers.

EDIT:

AWS support also informed me that eksctl has a `maxPodsPerNode` field, which is nice for anyone using eksctl and not Terraform like I am. eksctl takes a similar approach to the workaround above, except that it modifies kubelet-config.json instead of eni-max-pods.txt in a user data script, which might slightly more resilient to future EKS AMI changes than the approach above:

eksctl generated user data:

```
#!/bin/sh
set -ex
sed -i -E "s/^USE_MAX_PODS=\"\\$\{USE_MAX_PODS:-true}\"/USE_MAX_PODS=false/" /etc/eks/bootstrap.sh KUBELET_CONFIG=/etc/kubernetes/kubelet/kubelet-config.json
echo "$(jq ".maxPods=110" $KUBELET_CONFIG)" > $KUBELET_CONFIG
--a9e27770d41a7cacaf36053bde3ae957fd8ddb7852f7427271045222236e--
```

Final merged result:

```
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=a9e27770d41a7cacaf36053bde3ae957fd8ddb7852f7427271045222236e

--a9e27770d41a7cacaf36053bde3ae957fd8ddb7852f7427271045222236e
Content-Type: text/x-shellscript
Content-Type: charset="us-ascii"

#!/bin/sh
set -ex
sed -i -E "s/^USE_MAX_PODS=\"\\$\{USE_MAX_PODS:-true}\"/USE_MAX_PODS=false/" /etc/eks/bootstrap.sh KUBELET_CONFIG=/etc/kubernetes/kubelet/kubelet-config.json
echo "$(jq ".maxPods=110" $KUBELET_CONFIG)" > $KUBELET_CONFIG --a9e27770d41a7cacaf36053bde3ae957fd8ddb7852f7427271045222236e
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
B64_CLUSTER_CA=LS0tLS1CRUdJT
API_SERVER_URL=https://24369FE54FC4.gr7.us-west-2.eks.amazonaws.com
K8S_CLUSTER_DNS_IP=10.100.0.10
/etc/eks/bootstrap.sh lindarr --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/cluster-name=lindarr,alpha.eksctl.io/nodegroup-name=managed-ng-public2,eks.amazonaws.com/nodegroup-image=ami-0af965363397f19f5,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=managed-ng-public2,eks.amazonaws.com/sourceLaunchTemplateId=lt-0dfdbbe833f3c798e' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP
```

Start by reviewing the EKS managed worker Node Group API and its launch-template and user-data merge behavior. Then read the referenced amazon-eks-ami files, especially files/bootstrap.sh and files/eni-max-pods.txt, along with the documented workaround. Done means managed workers can accept bootstrap arguments without requiring an explicit AMI while retaining normal latest-AMI upgrade behavior.

Written by the indexing model from the issue text.