[sysbox-deploy-k8s] Authentication issue preventing ECR utilization in EKS clusters

In K8s scenarios where private registries are utilized to access container images, there's usually a need to authenticate the user through the mechanisms put in place by the cloud vendor of choice. This is typically the case In EKS clusters created through the eksctl tool, where kubelet is configured to fetch the pause image from a local Elastic-Container-Registry (ECR).

As part of sysbox-deploy-k8s' daemonset execution, there's logic to extract these configuration elements from kubelet and set them accordingly in cri-o's configuration (right, there's some overlap between cri-o and kubelet in regards to a few config attribs such as 'pause-image', 'pause-image-file-authentication', etc).

Problem with this approach is that we need to configure cri-o to be able to authenticate against ECR servers, and for that to happen we need to rely on AWS CLI tool, which isn't at hand when running within the context of sysbox-deploy-k8's daemonset. In consequence, no pod is able to initialize as can be seen below:

May 09 06:54:41 ip-192-168-14-235 crio[125942]: time="2022-05-09 06:54:41.510707022Z" level=info msg="RunSandbox: releasing container name: k8s_POD_aws-node-5lk92_kube-system_c1678921-5af4-47a0-9632-1e840836e172_0" id=0ec41843-885d-4e91-b635-241d8626fc2c name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
May 09 06:54:41 ip-192-168-14-235 crio[125942]: time="2022-05-09 06:54:41.510780796Z" level=info msg="RunSandbox: releasing container name: k8s_POD_aws-node-5lk92_kube-system_c1678921-5af4-47a0-9632-1e840836e172_0" id=0ec41843-885d-4e91-b635-241d8626fc2c name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
May 09 06:54:41 ip-192-168-14-235 kubelet-eks.daemon[126005]: E0509 06:54:41.511103  126005 remote_runtime.go:116] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = error creating pod sandbox with name \"k8s_aws-node-5lk92_kube-system_c1678921-5af4-47a0-9632-1e840836e172_0\": Error initializing source docker://602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.1-eksbuild.1: Error reading manifest 3.1-eksbuild.1 in 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause: unauthorized: authentication required"

Notice that kubelet is properly configured to make use of cri-o as its CRI, and the pause attributes are also defined as expected. However, there's some extra tweaking required to allow ECR authentication, probably through the utilization of
the pause_image_auth_file crio config attribute -- the pending question here is how to populate that file with the limited resources that are available when running within the context of sysbox-deploy-k8s daemonset?

ubuntu@ip-192-168-14-235:~$ sudo snap get kubelet-eks
Key                           Value
address                       0.0.0.0
anonymous-auth                false
args                          --node-labels=alpha.eksctl.io/nodegroup-name=ubuntu-nodes,alpha.eksctl.io/cluster-name=my-cluster,node-lifecycle=on-demand,alpha.eksctl.io/instance-id=i-06305353ab3dff91c
authentication-token-webhook  true
authorization-mode            Webhook
cgroup-driver                 cgroupfs
client-ca-file                /etc/kubernetes/pki/ca.crt
cloud-provider                aws
cluster-dns                   10.100.0.10
cluster-domain                cluster.local
cni-bin-dir                   /opt/cni/bin
cni-conf-dir                  /etc/cni/net.d
config                        /etc/kubernetes/kubelet/kubelet-config.json
container-runtime             remote
container-runtime-endpoint    unix:///var/run/crio/crio.sock
feature-gates                 RotateKubeletServerCertificate=true
kubeconfig                    /var/lib/kubelet/kubeconfig
max-pods                      29
network-plugin                cni
node-ip                       192.168.14.235
pod-infra-container-image     602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.1-eksbuild.1
register-node                 true
resolv-conf                   /run/systemd/resolve/resolv.conf