[EKS] [Fargate] [request]: Make fargate-scheduler retry pending pods

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
Make the fargate-scheduler retry / reconcile pending pods instead of trying it only once.

**Which service(s) is this request for?**
EKS, Fargate

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
My situation: I had a EKS Fargate profile with some pods running on it. I noticed that I configured the profile to use the wrong subnets. I wanted to change that. As of the AWS docs, Fargate profiles cannot be changed, but must be replaced:

>Fargate profiles can’t be changed. However, you can create a new updated profile to replace an existing profile, and then delete the original.
>
> >**Note**
> >Any Pods that are running using a Fargate profile are stopped and put in Pending state when the profile is deleted.
>
>If any Fargate profiles in a cluster are in the `DELETING` status, you must wait until after the Fargate profile is deleted before you create other profiles in that cluster.

From https://docs.aws.amazon.com/eks/latest/userguide/fargate-profile.html

I did this with terraform, and the provider (correctly) said the Fargate profile must be replaced. It tried evicting the respective pods, fails for some as there a PDB in place, and forcefully deleted the pods after a short while. Then, terraform created the new Fargate profile (identical settings except from the subnets). My pods failed to be scheduled with the warning `Misconfigured Fargate Profile: pod has an unknown fargate profile `, logged by `fargate-scheduler` as Kubernetes event. I waited for ~5min to check if reconciliation fixed it automatically, but it didn't and the pods remained in Pending state.

One of my apps running on Fargate is Karpenter. When using spot instances, it has 2min time from receiving an interrupt event to completely clean up the node, and spaws a new node. Karpenter pods not coming up again puts my clusters at high risk, as it can degrade almost every service I run.

**Are you currently working around this issue?**
I manually trigger pod recreation. This kicks off a new scheduling process and then it works fine, without any config change.

**Additional context**
This looks like the Fargate scheduler works event-based and does not reconcile. While/after the old Fargate profile is in DELETING state, Kubernetes creates new pods quickly, faster than the new Fargate profile is created (I think that's independent of the deployment method like manual/terraform/pulumi/crossplane/...). So the warning is likely correct and from the point in time when the new pods are already created but the Fargate profile is still in the fly. I think the problem is that the fargate-scheduler does not retry to schedule the pods.

I cannot simply create a second Fargate profile and move my pods before deleting the old profile, as the deployment config is part of a (internal) product with release cycles, but the subnets are an input given by my (internal) customers. The lifecycle of these two is very different.

Also, my (dozens of) affected clusters are operated by different teams, running in different version of the aforementioned internal product. The workaround of manual pod recreation was fine for investigating, but this is no reliable solution at scale.
I probably need to build some hacky automated workaround or put high invest to build a more sophisticated workaround for this. Reconciliation is one of the fundamental concepts of Kubernetes and AWS is very aware that Fargate profiles cannot be changed but must be replaced.

My clear expectation was that the Fargate scheduler reconciled pods the pods come up automatically after a short downtime.

Question / side quest: There's no way to see the logs of the fargate scheduler, is there? I have control plane logging for `scheduler` enabled, but it only writes `kube-scheduler` logs to CloudWatch. Given that the config value is `scheduler` and not `kube-scheduler`, I would expect that all AWS-deployed schedulers are affected by this setting. https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html

**Attachments**

Redacted Karpenter deployment resource

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: karpenter
namespace: "karpenter"
labels:
helm.sh/chart: karpenter-1.0.8
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
app.kubernetes.io/version: "1.0.8"
app.kubernetes.io/managed-by: Helm
security...com/part-of: -eks
annotations:
downscaler/downtime-replicas: "1"
argocd.argoproj.io/sync-options: Replace=true
configmap.reloader.stakater.com/reload: "karpenter-global-settings"
test...com/skip-nonpreempting-priorityclass: "true"
test...com/skip-no-cpu-resource-limit: "true"
spec:
replicas: 2
revisionHistoryLimit: 10
strategy:
rollingUpdate:
maxUnavailable: 1
type: RollingUpdate
selector:
matchLabels:
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
template:
metadata:
labels:
app.kubernetes.io/name: karpenter
app.kubernetes.io/instance: karpenter
security...com/part-of: -eks
..com/run-on: fargate
eks.amazonaws.com/fargate-profile: -eks
spec:
serviceAccountName: karpenter
securityContext:
fsGroup: 65532
priorityClassName: "system-cluster-critical"
dnsPolicy: None
dnsConfig:
nameservers:
- 100.78.0.2
containers:
- name: controller
securityContext:
runAsUser: 65532
runAsGroup: 65532
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
image: ".dkr.ecr.eu-central-1.amazonaws.com/-eks/prod/karpenter/controller:v1.0.8-pr775"
imagePullPolicy: IfNotPresent
env: [...]
ports:
- name: http-metrics
containerPort: 8080
protocol: TCP
- name: webhook-metrics
containerPort: 8001
protocol: TCP
- name: https-webhook
containerPort: 8443
protocol: TCP
- name: http
containerPort: 8081
protocol: TCP
livenessProbe:
initialDelaySeconds: 30
timeoutSeconds: 30
httpGet:
path: /healthz
port: http
readinessProbe:
initialDelaySeconds: 5
timeoutSeconds: 30
httpGet:
path: /readyz
port: http
resources:
limits:
cpu: 449m
memory: 765Mi
requests:
cpu: 449m
memory: 765Mi
nodeSelector:
kubernetes.io/os: linux
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: karpenter.sh/nodepool
operator: DoesNotExist
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values:
- karpenter
topologyKey: topology.kubernetes.io/zone
topologySpreadConstraints:
- labelSelector:
matchLabels:
app.kubernetes.io/instance: karpenter
app.kubernetes.io/name: karpenter
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
tolerations:
- key: CriticalAddonsOnly
operator: Exists
schedulerName: fargate-scheduler
```

The issue names the fargate-scheduler entry point and the schedulerName field, but no source files or tests are identified. Start by locating the scheduler implementation and its handling of the `Misconfigured Fargate Profile` event; done means pending pods are retried or reconciled after a replacement profile becomes available, without manual pod recreation.

Written by the indexing model from the issue text.