KEP-2903: Support Elastic PyTorch in TrainJob

### What you would like to be added?

As part of [Kubeflow Trainer API](https://github.com/kubeflow/trainer/tree/master/docs/proposals/2170-kubeflow-trainer-v2), we designed [the ElasticPolicy API](https://github.com/kubeflow/trainer/blob/master/pkg/apis/trainer/v1alpha1/trainingruntime_types.go#L219) which should allow users to run PyTorch in elastic mode: https://docs.pytorch.org/docs/stable/elastic/run.html

It would be nice if someone could drive KEP for that, and propose API changes to the next release: Trainer v2.2

```yaml
elasticPolicy:
minNodes: 2
maxNodes: 5
metrics:
- type: Resource
resource:
name: nvidia.com/gpu
target:
type: Utilization
averageUtilization: 75
```

Potentially, we should support elastic JobSet for that: https://github.com/kubernetes-sigs/jobset/issues/463

cc @kubeflow/kubeflow-trainer-team

/area api
/area controller
/help

### Why is this needed?

We should support Elastic TrainJobs like in Training Operator V1 PyTorchJob.

### Love this feature?

Give it a 👍 We prioritize the features with most 👍