Extend MPI Plugin with MPICH and Intel Backends

### What you would like to be added?

During the latest Trainer call some users express interest of supporting additional backends in MPI plugin to run some HPC jobs with TrainJobs.
That will also ensure we have 1-1 parity with MPI Operator: https://youtu.be/SQGjGTvjObo?t=1407

The goal is to [extend the `MPIImplementation`](https://github.com/kubeflow/trainer/blob/master/pkg/apis/trainer/v1alpha1/trainingruntime_types.go#L311) to support required backends.

cc @vsoch @kannon92 @hahahannes @eero-t @kubeflow/kubeflow-trainer-team @robert-bell @eero-t @killianmuldoon @catblade

### Why is this needed?

Enhance MPI plugin with IntelMPI and MPICH backends

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Start with the MPIImplementation definition in pkg/apis/trainer/v1alpha1/trainingruntime_types.go and inspect the MPI plugin's existing backend handling. Add IntelMPI and MPICH support so the MPI plugin supports the requested backends and reaches parity with the MPI Operator.

Written by the indexing model from the issue text.