Occasional "exec format error"

**Image I'm using:**
AMI: `bottlerocket-aws-k8s-1.22-x86_64-v1.9.0-159e4ced`
EKS:

```
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.10-eks-84b4fe6", GitCommit:"cc6a1b4915a99f49f5510ef0667f94b9ca832a8a", GitTreeState:"clean", BuildDate:"2022-06-09T18:24:04Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
```

Occasionally containers are refusing to start, for example:

```
➜ ~ k -n argocd logs argocd-server-67f9f4788c-jx6nv
exec /usr/local/bin/argocd-server: exec format error

➜ ~ k -n argocd get pod -o wide | grep argocd-server
argocd-server-67f9f4788c-jx6nv 0/1 CrashLoopBackOff 28 (2m25s ago) 120m 10.98.70.121 ip-10-98-70-152.eu-west-1.compute.internal
argocd-server-67f9f4788c-nnxxc 1/1 Running 0 17h 10.98.60.136 ip-10-98-60-34.eu-west-1.compute.internal

➜ ~ k -n argocd get pod -l app.kubernetes.io/name=argocd-server -o yaml | grep image
image: quay.io/argoproj/argocd:v2.4.7
imagePullPolicy: Always
image: quay.io/argoproj/argocd:v2.4.7
imageID: quay.io/argoproj/argocd@sha256:f887f854ab22f7f29f915aae2b765f2948d1555d61e9ce3ca9e659f8df22ab2b
image: quay.io/argoproj/argocd:v2.4.7
imagePullPolicy: Always
image: quay.io/argoproj/argocd:v2.4.7
imageID: quay.io/argoproj/argocd@sha256:f887f854ab22f7f29f915aae2b765f2948d1555d61e9ce3ca9e659f8df22ab2b

➜ ~ k describe node ip-10-98-70-152.eu-west-1.compute.internal ip-10-98-60-34.eu-west-1.compute.internal
Name: ip-10-98-70-152.eu-west-1.compute.internal
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=m5a.large
beta.kubernetes.io/os=linux
bottlerocket.aws/updater-interface-version=2.0.0
cni=cilium
eks.amazonaws.com/capacityType=SPOT
eks.amazonaws.com/nodegroup=sta-0c008cd98efe613ce-20220812085801371900000017
eks.amazonaws.com/nodegroup-image=ami-05ef4fee9a548939d
eks.amazonaws.com/sourceLaunchTemplateId=lt-0e03ae3eb8ac34efd
eks.amazonaws.com/sourceLaunchTemplateVersion=2
failure-domain.beta.kubernetes.io/region=eu-west-1
failure-domain.beta.kubernetes.io/zone=eu-west-1c
k8s.io/cloud-provider-aws=2a1dc32348e49d8c3d205b11025253c8
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-10-98-70-152.eu-west-1.compute.internal
kubernetes.io/os=linux
node.kubernetes.io/instance-type=m5a.large
topology.ebs.csi.aws.com/zone=eu-west-1c
topology.kubernetes.io/region=eu-west-1
topology.kubernetes.io/zone=eu-west-1c
Annotations: csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-0059589916b9e311c"}
io.cilium.network.ipv4-cilium-host: 10.98.70.90
io.cilium.network.ipv4-health-ip: 10.98.70.88
io.cilium.network.ipv4-pod-cidr: 10.152.0.0/16
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 15 Aug 2022 21:06:22 +0200
Taints:
Unschedulable: false
Lease:
HolderIdentity: ip-10-98-70-152.eu-west-1.compute.internal
AcquireTime:
RenewTime: Tue, 16 Aug 2022 14:06:34 +0200
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
ReadonlyFilesystem False Tue, 16 Aug 2022 14:03:31 +0200 Mon, 15 Aug 2022 21:11:35 +0200 FilesystemIsNotReadOnly Filesystem is not read-only
KernelDeadlock False Tue, 16 Aug 2022 14:03:31 +0200 Mon, 15 Aug 2022 21:11:35 +0200 KernelHasNoDeadlock kernel has no deadlock
NetworkUnavailable False Mon, 15 Aug 2022 21:07:00 +0200 Mon, 15 Aug 2022 21:07:00 +0200 CiliumIsUp Cilium is running on this node
MemoryPressure False Tue, 16 Aug 2022 14:02:10 +0200 Mon, 15 Aug 2022 21:06:21 +0200 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 16 Aug 2022 14:02:10 +0200 Mon, 15 Aug 2022 21:06:21 +0200 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 16 Aug 2022 14:02:10 +0200 Mon, 15 Aug 2022 21:06:21 +0200 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 16 Aug 2022 14:02:10 +0200 Tue, 16 Aug 2022 11:32:04 +0200 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.98.70.152
Hostname: ip-10-98-70-152.eu-west-1.compute.internal
InternalDNS: ip-10-98-70-152.eu-west-1.compute.internal
Capacity:
attachable-volumes-aws-ebs: 25
cpu: 2
ephemeral-storage: 41261776Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7877168Ki
pods: 29
Allocatable:
attachable-volumes-aws-ebs: 25
cpu: 1930m
ephemeral-storage: 36953110875
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7186992Ki
pods: 29
System Info:
Machine ID: ec294af3fb7450c1a90fe38fb1e8d2e9
System UUID: ec294af3-fb74-50c1-a90f-e38fb1e8d2e9
Boot ID: f9433ce1-8279-403d-9006-30f9e7607973
Kernel Version: 5.10.130
OS Image: Bottlerocket OS 1.9.0 (aws-k8s-1.22)
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.6+bottlerocket
Kubelet Version: v1.22.10-eks-7dc61e8
Kube-Proxy Version: v1.22.10-eks-7dc61e8
ProviderID: aws:///eu-west-1c/i-0059589916b9e311c
Non-terminated Pods: (24 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
argocd argocd-redis-ha-haproxy-5b46945689-f4j5s 10m (0%) 0 (0%) 100Mi (1%) 256Mi (3%) 14h
argocd argocd-redis-ha-server-1 20m (1%) 0 (0%) 96Mi (1%) 320Mi (4%) 16h
argocd argocd-server-67f9f4788c-jx6nv 10m (0%) 0 (0%) 64Mi (0%) 256Mi (3%) 121m
aws-ebs-csi-driver ebs-csi-node-fm96d 30m (1%) 0 (0%) 36M (0%) 387M (5%) 16h
backoffice-cm backoffice-cm-deployment-64b7d7f579-56qvg 500m (25%) 0 (0%) 1Gi (14%) 2Gi (29%) 5h6m
brupop-bottlerocket-aws brupop-agent-f45ps 10m (0%) 0 (0%) 34M (0%) 45M (0%) 16h
brupop-bottlerocket-aws brupop-apiserver-54f6795bd-4256b 10m (0%) 0 (0%) 34M (0%) 45M (0%) 14h
brupop-bottlerocket-aws brupop-controller-deployment-8c778d976-w64db 10m (0%) 0 (0%) 34M (0%) 45M (0%) 14h
cert-manager cert-manager-cainjector-6f7597f6f4-hgwgv 10m (0%) 0 (0%) 76M (1%) 1565M (21%) 14h
cilium cilium-csl7q 100m (5%) 0 (0%) 100Mi (1%) 0 (0%) 154m
csi-snapshots csi-snapshotter-0 30m (1%) 0 (0%) 72M (0%) 294M (3%) 14h
goldilocks goldilocks-controller-57d695bdf5-tzcj2 22m (1%) 321m (16%) 161Mi (2%) 1503Mi (21%) 14h
goldilocks goldilocks-dashboard-6b5b46dbc8-99z8t 10m (0%) 101m (5%) 34Mi (0%) 218Mi (3%) 14h
goldilocks vpa-recommender-6b97cd4fb4-vx8bm 10m (0%) 101m (5%) 49Mi (0%) 458Mi (6%) 14h
ingress-nginx-internal ingress-nginx-controller-7764f4fddd-2q56k 10m (0%) 0 (0%) 108M (1%) 1104M (15%) 14h
keda keda-metrics-apiserver-6c487b448d-p2mnk 100m (5%) 0 (0%) 100Mi (1%) 1000Mi (14%) 14h
kube-system coredns-5947f47f5f-94t4s 100m (5%) 0 (0%) 70Mi (0%) 170Mi (2%) 14h
kube-system kube-proxy-7xmrt 100m (5%) 0 (0%) 0 (0%) 0 (0%) 17h
kube-system metrics-server-848988b695-wm9m5 10m (0%) 0 (0%) 36M (0%) 834M (11%) 16h
node-problem-detector node-problem-detector-5556g 10m (0%) 0 (0%) 34M (0%) 1122M (15%) 16h
polaris polaris-dashboard-c696fb87-j9g7s 100m (5%) 0 (0%) 128Mi (1%) 512Mi (7%) 5h6m
postgres-operator ext-postgres-operator-dbdb98546-56xg5 10m (0%) 0 (0%) 34M (0%) 246M (3%) 5h6m
prometheus prometheus-prometheus-node-exporter-g9wmh 10m (0%) 0 (0%) 16Mi (0%) 32Mi (0%) 16h
promtail promtail-q79pc 100m (5%) 0 (0%) 64Mi (0%) 512Mi (7%) 16h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 1332m (69%) 523m (27%)
memory 2601443456 (35%) 13325876160 (181%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events:

Name: ip-10-98-60-34.eu-west-1.compute.internal
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=m5.large
beta.kubernetes.io/os=linux
bottlerocket.aws/updater-interface-version=2.0.0
cni=cilium
eks.amazonaws.com/capacityType=SPOT
eks.amazonaws.com/nodegroup=sta-0566348ba06653b8a-20220812085801373500000019
eks.amazonaws.com/nodegroup-image=ami-05ef4fee9a548939d
eks.amazonaws.com/sourceLaunchTemplateId=lt-08fd57448fa5945b5
eks.amazonaws.com/sourceLaunchTemplateVersion=2
failure-domain.beta.kubernetes.io/region=eu-west-1
failure-domain.beta.kubernetes.io/zone=eu-west-1b
k8s.io/cloud-provider-aws=2a1dc32348e49d8c3d205b11025253c8
kubernetes.io/arch=amd64
kubernetes.io/hostname=ip-10-98-60-34.eu-west-1.compute.internal
kubernetes.io/os=linux
node.kubernetes.io/instance-type=m5.large
topology.ebs.csi.aws.com/zone=eu-west-1b
topology.kubernetes.io/region=eu-west-1
topology.kubernetes.io/zone=eu-west-1b
Annotations: csi.volume.kubernetes.io/nodeid: {"ebs.csi.aws.com":"i-052270355dc863cb5"}
io.cilium.network.ipv4-cilium-host: 10.98.60.134
io.cilium.network.ipv4-health-ip: 10.98.60.172
io.cilium.network.ipv4-pod-cidr: 10.34.0.0/16
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 15 Aug 2022 20:59:53 +0200
Taints:
Unschedulable: false
Lease:
HolderIdentity: ip-10-98-60-34.eu-west-1.compute.internal
AcquireTime:
RenewTime: Tue, 16 Aug 2022 14:06:38 +0200
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
KernelDeadlock False Tue, 16 Aug 2022 14:04:50 +0200 Mon, 15 Aug 2022 21:02:53 +0200 KernelHasNoDeadlock kernel has no deadlock
ReadonlyFilesystem False Tue, 16 Aug 2022 14:04:50 +0200 Mon, 15 Aug 2022 21:02:53 +0200 FilesystemIsNotReadOnly Filesystem is not read-only
NetworkUnavailable False Mon, 15 Aug 2022 21:00:28 +0200 Mon, 15 Aug 2022 21:00:28 +0200 CiliumIsUp Cilium is running on this node
MemoryPressure False Tue, 16 Aug 2022 14:05:00 +0200 Mon, 15 Aug 2022 21:02:47 +0200 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 16 Aug 2022 14:05:00 +0200 Mon, 15 Aug 2022 21:02:47 +0200 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 16 Aug 2022 14:05:00 +0200 Mon, 15 Aug 2022 21:02:47 +0200 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 16 Aug 2022 14:05:00 +0200 Tue, 16 Aug 2022 11:32:24 +0200 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.98.60.34
Hostname: ip-10-98-60-34.eu-west-1.compute.internal
InternalDNS: ip-10-98-60-34.eu-west-1.compute.internal
Capacity:
attachable-volumes-aws-ebs: 25
cpu: 2
ephemeral-storage: 41261776Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7918136Ki
pods: 29
Allocatable:
attachable-volumes-aws-ebs: 25
cpu: 1930m
ephemeral-storage: 36953110875
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 7227960Ki
pods: 29
System Info:
Machine ID: ec2b47de995c9eb1d1e7be45efe6de37
System UUID: ec2b47de-995c-9eb1-d1e7-be45efe6de37
Boot ID: 66b5b8bd-d9d7-4a44-b4e1-4500dd7f25bd
Kernel Version: 5.10.130
OS Image: Bottlerocket OS 1.9.0 (aws-k8s-1.22)
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.6+bottlerocket
Kubelet Version: v1.22.10-eks-7dc61e8
Kube-Proxy Version: v1.22.10-eks-7dc61e8
ProviderID: aws:///eu-west-1b/i-052270355dc863cb5
Non-terminated Pods: (28 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
argocd argocd-applicationset-controller-84c7fbf59c-d7cch 10m (0%) 0 (0%) 50Mi (0%) 100Mi (1%) 17h
argocd argocd-redis-ha-haproxy-5b46945689-4l8px 10m (0%) 0 (0%) 100Mi (1%) 256Mi (3%) 17h
argocd argocd-repo-server-6ff96df8d6-vxc7f 500m (25%) 0 (0%) 128Mi (1%) 1536Mi (21%) 14h
argocd argocd-server-67f9f4788c-nnxxc 10m (0%) 0 (0%) 64Mi (0%) 256Mi (3%) 17h
aws-ebs-csi-driver ebs-csi-controller-7d7dbd98-kg6vj 60m (3%) 0 (0%) 96M (1%) 776M (10%) 17h
aws-ebs-csi-driver ebs-csi-node-zwlll 30m (1%) 0 (0%) 36M (0%) 387M (5%) 17h
backoffice-cm kafka-connect-deployment-c79668bc6-x9fdr 0 (0%) 0 (0%) 0 (0%) 0 (0%) 17h
brupop-bottlerocket-aws brupop-agent-gtdvt 10m (0%) 0 (0%) 34M (0%) 45M (0%) 17h
brupop-bottlerocket-aws brupop-apiserver-54f6795bd-gh7q7 10m (0%) 0 (0%) 34M (0%) 45M (0%) 17h
cert-manager cert-manager-788b7f5656-gnz6f 10m (0%) 0 (0%) 35M (0%) 727M (9%) 17h
cert-manager cert-manager-webhook-8664f454c9-4z5nm 10m (0%) 0 (0%) 34M (0%) 472M (6%) 14h
cilium cilium-45tn7 100m (5%) 0 (0%) 100Mi (1%) 0 (0%) 154m
cilium hubble-ui-5d9b569687-f4rcw 0 (0%) 0 (0%) 0 (0%) 0 (0%) 17h
csi-snapshots snapshot-controller-65b949c64c-v92dq 10m (0%) 0 (0%) 34M (0%) 98M (1%) 14h
ingress-nginx ingress-nginx-controller-6755cfb894-8kvl5 10m (0%) 0 (0%) 108M (1%) 1104M (14%) 17h
keda keda-metrics-apiserver-6c487b448d-wr828 100m (5%) 0 (0%) 100Mi (1%) 1000Mi (14%) 17h
keda keda-operator-8449c9c8b8-mmzx6 100m (5%) 0 (0%) 100Mi (1%) 1000Mi (14%) 17h
kube-downscaler kube-downscaler-86cbf4f595-h28dv 10m (0%) 0 (0%) 49M (0%) 459M (6%) 17h
kube-system cluster-autoscaler-5fb96c46c7-chhjb 10m (0%) 0 (0%) 125M (1%) 1482M (20%) 17h
kube-system coredns-5947f47f5f-ks6zp 100m (5%) 0 (0%) 70Mi (0%) 170Mi (2%) 17h
kube-system descheduler-d78748c74-6j98w 10m (0%) 0 (0%) 34Mi (0%) 1Gi (14%) 17h
kube-system kube-proxy-w789r 100m (5%) 0 (0%) 0 (0%) 0 (0%) 17h
kube-system metrics-server-848988b695-7tmzv 10m (0%) 0 (0%) 36M (0%) 834M (11%) 17h
node-problem-detector node-problem-detector-8jzhx 10m (0%) 0 (0%) 34M (0%) 1122M (15%) 17h
polaris polaris-dashboard-c696fb87-55hsd 100m (5%) 0 (0%) 128Mi (1%) 512Mi (7%) 122m
prometheus prometheus-prometheus-node-exporter-dwjz9 10m (0%) 0 (0%) 16Mi (0%) 32Mi (0%) 17h
prometheus prometheus-prometheus-prometheus-0 310m (16%) 200m (10%) 1138Mi (16%) 4402Mi (62%) 17h
promtail promtail-wxqph 100m (5%) 0 (0%) 64Mi (0%) 512Mi (7%) 17h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 1750m (90%) 200m (10%)
memory 2848620992 (38%) 18875620800 (255%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events:
```

Happens randomly to various containers, like prometheus etc. No, I'm not trying to run amd64 compiled ones.

**How to reproduce the problem:**
Honestly can not figure this out, happens randomly. Thought it might be related to kubearmor or cilium (IPAM mode) but its not.

**NOTE** Left clusters running on AL2 image for few days, never happened.

Any help appreciated.