Automatic Failover for Inference Workloads upon GPU Failure Detection

### Main idea

## References

- [MS Teams Thread](https://teams.microsoft.com/l/message/19:3c0acc0825024daeb2211f55c4e7f4aa@thread.skype/1730286780612?tenantId=13c6a44d-9b52-4b9e-aa34-0513ee7131f2&groupId=74ae2c4d-ec4d-4fdf-b2c2-f5041d1e8631&parentMessageId=1730286780612&teamName=devops&channelName=Backend.AI%20Talks&createdTime=1730286780612)
## Motivation

Unlike batch workloads, inference/service workloads must remain operational for the duration of the session.To ensure session continuity, inference/service workloads should include a retry mechanism that enables failover if they encounter issues.If recovery is not possible even with the retry policy, metrics such as RetryCount will be used to indicate abnormal situations.

## Main Tasks

I think three main tasks need to be performed:

1. Implement probe actions to periodically check for normal operation.
1. Manage retry metrics and implement actions for the retry policy.
1. Implement a dashboard for visualizing metrics and alerts for notifications.
## Expected Results

1. Issues that can be resolved with simple restarts will likely recover on their own.
1. It will be possible to quickly identify when problems occur.
### Alternative ideas

_No response_

### Anything else?

_No response_

JIRA Issue: BA-71