Use Prometheus reasonably to avoid unnecessary job failed.

What would you like to be added:
Use Prometheus reasonably to avoid unnecessary job failed.
Why is this needed:
At present, the Prometheus system is only used as an alarm system, and its capabilities have not been fully utilized. Consider a situation where a machine has 8 GPU cards, but one of the cards was sent back to the factory for repair. At this time, the Prometheus system will report the change in the number of GPUs, but the task management system is not aware of this problem. At this time, if there are already 7 GPU cards in the working state, and the user is still submitting tasks. The task management system will continue to apply for the 8th GPU card from the node, instead of keeping the task in a waiting state, the task will fail at this time.
Without this feature, how does the current module work:
When the number of GPUs changes, the task often failed.
Components that may involve changes:

Trace how Prometheus reports GPU-count changes and how the task management system requests GPUs from a node. Define the behavior for a reduced GPU count so tasks remain waiting instead of failing, then verify the scheduling flow handles that condition.