Transparent GPU Checkpointing of TrainJobs with CRIU

### What you would like to be added?

As part of the recent Training WG call, we discussed transparent GPU checkpointing for AI workloads using CRIU.
Recordings: https://youtu.be/j6BR7VEcWPY?t=559

The initial discussions started in this issue, but we want to separate the exporter Job from the transparent checkpointing functionality: https://github.com/kubeflow/trainer/issues/2245
Let’s use this issue to track progress and explore how we can integrate this capability with Kubeflow Trainer.

Resources:
- KubeCon London talks: https://youtu.be/BSoEY_tpxIo and https://youtu.be/3oWODC2mdk0
- CRIU Coordinator: https://github.com/checkpoint-restore/criu-coordinator
- Checkpointing ctl: https://github.com/checkpoint-restore/checkpointctl

cc @kubeflow/kubeflow-trainer-team @kramaranya @astefanutti @rst0git @adrianreber @shravan-achar @bigsur0

### Why is this needed?

Enhance TrainJob fault-tolerance and resiliency with transparent GPU checkpointing.

### Love this feature?

Give it a 👍 We prioritize the features with most 👍