[EKS] Managed Nodegroups IP allocation issue with two attached subnets spanning same AZ

We have encountered an issue where we have a Managed Node-group attached with a subnet (lets say subnet1 spanning AZ1) has emptied with addressable IP's for pods & nodes. We to workaround this issue carved a new subnet (lets say subnet2 spanning AZ1) and created a new managed nodegroup attached with both subnets (subnet1 & subnet2 both spanning AZ1).

Now when we have any new workload dropped targetting this nodegroup we run into issues where pods are unable to fetch IP because it is trying to get one from subnet1 and ignore the existence of subnet2 entirely. After a couple of cleanup and retries the pods finally see the subnet2 and fetch IP from their. The issue possibly is the scheduler do not have the capability to understand if it cannot find IP to attached subnet of the managed nodegroup it can look for any other subnet if exist.

Error we see :

Normal Scheduled 52s default-scheduler Successfully assigned xxxxxx-design-sd/sd-2 to ip-xx-xx-xx-xx.us-west-2.compute.internal
Warning FailedCreatePodSandBox 41s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container

The report names managed nodegroups, the Kubernetes scheduler, and kubelet's FailedCreatePodSandBox event, but no repository files or tests. Start by reproducing the workload on a nodegroup using two subnets in the same AZ and inspect the pod and networking events. Done means explaining the subnet selection and establishing the expected fallback behavior when the first subnet has no available IPs.

Written by the indexing model from the issue text.