KEP-3562: Handle TrainJob failures and invalid search space tracking

### What you would like to be added?

**Context:**
During the initial implementation of [KEP-3562](https://github.com/kubeflow/trainer/pull/3565), we [discussed](https://github.com/kubeflow/trainer/pull/3565#discussion_r3579995629) how to handle individual `TrainJob` failures without failing the entire `OptimizationJob`.

**Problem:**
It is difficult to distinguish between failures caused by invalid hyperparameters (e.g., CUDA OOM) and transient infrastructure issues (e.g., node network failures). Currently, failed trials do not pass penalty information back to the optimizer.

**Future Goals (Phase 2):**
* Ensure the `OptimizationJob` continues running even if individual `TrainJobs` fail.
* Investigate passing `inf` values or using `constraints_func` to tell the Optimizer to avoid suggesting invalid parameters.
* Explore mechanisms (e.g., `PodFailurePolicies` or user-code signaling) to differentiate infrastructure errors from configuration errors.

*Relates to KEP-3562 Master Tracking [Issue](https://github.com/kubeflow/trainer/issues/3562).*

### Why is this needed?

To prevent a single bad hyperparameter trial from crashing the entire experiment, and to teach the optimizer to avoid bad configurations by explicitly penalizing them instead of confusing them with random infrastructure glitches.

### Love this feature?

Give it a 👍 We prioritize the features with most 👍