[EKS] [request]: Cluster init job to bootstrap new clusters

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
This is somewhat related to #923 (reliant on it actually) but I didn't want to muck up that feature request with additional noise since on its own its quite good and would be a great addition.

When provisioning new clusters, there are a number of steps that unfortunately have to be manually performed that are repetitious; especially when treating clusters as temporary/ephemeral (i.e. - blue/green cluster upgrades, sandbox clusters, etc.). An example would be:
1. Provision cluster and a managed node group (at least one node so that the core service pods come up happy and the provisioning succeeds. There is an alternate route where you don't deploy a node group yet, make the necessary changes, and then deploy your node groups. Its 6 vs half a dozen in the end though regarding which route you take)
2. Once the cluster and node group are deployed, the first step we'll look at is pod networking:
a. One route is to remove the VPC CNI permissions from the node group and rely on an [IRSA for providing the VPC CNI](https://github.com/terraform-aws-modules/terraform-aws-eks/blob/9a99689cc13147f4afc426b34ba009875a28614e/examples/eks_managed_node_group/main.tf#L41) the necessary permissions where the VPC CNI is a managed addon. I cannot launch a node group without the VPC CNI permissions and rely only on the IRSA role for the addon because its a chicken vs the egg problem. I have to provision first with the node having the permissions, then come back and remove and rely on the IRSA only
b. Another route is where I don't want to use the VPC CNI and I need to delete it and provision the desired CNI/configuration. Again, this can only be done manually since EKS makes the default assumption to use the VPC CNI (I assume for a good "out of the box" experience, which makes sense)
* Note: the other managed addons that are provided by default can also fall within a similar pattern. In my experience, folks want to codify what goes into their cluster. Trying to codify something that was provisioned outside of your control is one of the core issues here and why #923 is tied directly to this topic

3. Lastly, at this point we want to switch to our preferred cluster management solution which usually introduces another chicken vs. the egg situation. This primarily concerns folks who are following GitOps practices as well as folks who are trying to follow recommended practices by disabling the cluster control plane public endpoint.
- GitOps is great for clusters that do not have a publicly accessible endpoint; their operators sit within the cluster and pull changes in from external manifest repositories, image registries, etc. (pull method vs push)
- However, how do we first provision our GitOps operator to start this pull based process - I've seen hacks where clusters are provisioned with public endpoints enabled, push deploy the GitOps oeprator with a script, then remove the public endpoint access. Its a cumbersome process to say the least. This is where having a job that is again sitting within the cluster or at least within the same network (access) as the cluster, we can move that bootstrapping to this job that is executed once when creating a new cluster.
- Once the GitOps operator is provisioned, all changes can be handled via the codified manifests that the operator monitors for changes

#### ✨ The Request ✨

The ability to provide a cluster `init` job that is run once upon cluster creation (think user data, but for clusters).

The changes in #923 are required to make this work without race conditions. If default components are installed, its a race to get the default components removed from the cluster before nodes come up using them and the desired components installed (i.e. - CoreDNS, CNI, etc.). If they are not on the cluster to begin with, users are simply providing their desired configurations via the init job. The init job needs to run in a loop for those tools that are not designed as operators; but an operator (which itself is its own control loop) is the preferred approach. The longer form for ArgoCD is shown below - the preferred alternative would be to use the ArgoCD operator instead.

This init job also alleviates the shortcomings of running a Fargate only cluster where users have to [manually intervene to configure CoreDNS for Fargate](https://docs.aws.amazon.com/eks/latest/userguide/fargate-getting-started.html#fargate-gs-coredns)

This lifecycle roughly looks like the following:

```mermaid
stateDiagram-v2
[*] --> ControlPlane
ControlPlane --> InitJob
InitJob --> InitJobRetry
InitJobRetry --> InitJob
ControlPlane --> NodeGroup
ControlPlane --> FargateProfile
InitJob --> [*]
NodeGroup --> [*]
FargateProfile --> [*]
```

1. Users provision a bare cluster. When they submit the cluster creation, they have the ability to provide an init job spec; see a rough example below
2. The control plane will be provisioned as usual, VPC endpoints attached into client account, etc.
3. Following creation of the control plane, the init job is kicked off which runs asynchronously to node group/profile creation. Ideally, users should employ an operator pattern for their bootstrapping mechanism of choice which has its own self-contained control loop, but the init job itself can also be configured to backoff and retry `x` number of times for up to `y` timeout (standard job specs within the manifest). The key will be how to properly surface failures to users when the init job does not succeed - because nodes will most likely fail to join the cluster as well in that scenario.
4. Upon successful execution of the init job, nodes should join the cluster successfully and the cluster bootstrap process is complete. I am positive that there are other use cases that people employ in bootstrapping a fresh cluster; this is just the most common set of steps that I have encountered when creating clusters or helping clients with their clusters (i.e - what steps are folks taking for restoring persistent volumes or handling stateful sets, etc. when bootstrapping a new cluster)

```yaml
apiVersion: batch/v1
kind: Job
metadata:
name: clusterInit
spec:
template:
spec:
containers:
- name: al2
# most likely will need a new image which has kubectl installed by default to match cluster version
image: public.ecr.aws/amazonlinux/amazonlinux:2
command:
- /bin/sh
- -c
- |
# we will assume kubectl is installed either via the image or script
# and the pod will most likely require IRSA
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
ARGOCD_OPTS='--port-forward-namespace argocd'
ARGOCD_INIT_PWD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo)
ARGOCD_NEW_PWD=$(aws secretsmanager get-secret-value --secret-id prod/mysecret)
argocd account update-password --current-password $ARGOCD_INIT_PWD --new-password $ARGOCD_NEW_PWD
# Bootstrap cluster using app of apps pattern https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/
# One suite of apps could be management apps - CoreDNS, Cillium CNI, etc., another for monitoring/observability,
# andother for applications, etc. Up to the cluster administrator(s) to decide
argocd app create apps \
--dest-namespace argocd \
--dest-server https://kubernetes.default.svc \
--repo https://github.com/argoproj/argocd-example-apps.git \
--path apps
argocd app sync apps \
--async \ # async is important since it won't succeed until nodes are launched
--retry-backoff-duration 2m \ # add some backoff and retries to give time for nodes to launch
--retry-limit 5 \
--timeout 630
restartPolicy: Never
```

**Which service(s) is this request for?**
EKS

**Are you currently working around this issue?**
A mix of hacky scripts and manual intervention - "playbooks" if you will

**Additional context**
- https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1822
- https://github.com/aws/containers-roadmap/issues/923#issuecomment-907814157
- https://github.com/aws/containers-roadmap/issues/71
- https://github.com/aws/containers-roadmap/issues/1592
- https://github.com/aws/containers-roadmap/issues/1593

- https://github.com/awslabs/amazon-eks-ami/issues/117
- https://github.com/aws/amazon-vpc-cni-k8s/issues/214
- https://github.com/aws/amazon-vpc-cni-k8s/issues/176
- https://github.com/aws-quickstart/quickstart-amazon-eks/issues/51
- https://github.com/aws/containers-roadmap/issues/185 is tangentially related due to the various hacks that people have to employ to initially provision access into the cluster via a codified process. With the approach here, this can help alleviate some of those issues - instead of giving access to the role that created the cluster (usually a CI/CD role that is quite elevated and heavily protected), users can offload the aws-auth configmap to be managed by their GitOps operator and/or they can update the configmap upon cluster creation

No repository files, tests, or implementation entry points are named. Start by reading linked issue #923 and the related issues, then compare them with the proposed lifecycle and example init Job; done would require a decided, supported design for running and reporting a cluster-init job during EKS cluster creation.

Written by the indexing model from the issue text.