Untolerated Taint when InstaScale True

Describe the Bug

When instascale=True in cluster configuration and existing nodes have requisite resources, the pods are not being scheduled to to taint intolerance. I'm filing this bug here, but my guess is that this is a bug with InstaScale.

Codeflare Stack Component Versions

Codeflare SDK: 0.8.0
Codeflare Operator: 1.0.0-rc1
Other:

Steps to Reproduce the Bug

@chcost Could you please comment with your cluster configuration and available nodes?

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

What Have You Already Tried to Debug the Issue?

Scheduled the same AppWrapper with InstaScale set to false and it successfully deployed the KubeRay cluster without any issues.

Expected Behavior

InstaScale should prefer available if possible

Screenshots, Console Output, Logs, etc.

@chcost Here as well if you have any

Affected Releases

0.8.0, main

Additional Context

Add as applicable and when known:

• OS: 2
• OS Version: [e.g. RedHat Linux X.Y.Z, MacOS Monterey, ...]
• Browser (UI issues): 1) Chrome, 2) Safari, 3) Firefox, 4) Other (describe): [1 - 4 + description?]
• Browser Version (UI issues): [e.g. Firefix 97.0]
• Cloud: 1) AWS, 2) IBM Cloud, 3) Other (describe), or 4) on-premise: [1 - 4 + description?]
• Kubernetes: 1) OpenShift, 2) Other K8s [1 - 2 + description]
• OpenShift or K8s version: [e.g. 1.23.1]
• Other relevant info

Add any other information you think might be useful here.

Start by collecting the missing cluster configuration, available-node details, and logs requested in the issue. Reproduce the AppWrapper with instascale=true and false, then inspect the InstaScale and KubeRay scheduling behavior around taints and tolerations. Done means pods use suitable existing nodes without an intolerable taint preventing scheduling.