Integrate Kubeflow SDK with OpenTelemetry

### What you would like to be added?

We would like to explore opportunities to integrate Kubeflow SDK with Open Telemetry for better observability and telemetry of AI Workloads.
For example, we can take a look at OTel and GenAI integrations: https://opentelemetry.io/blog/2024/otel-generative-ai/

This was one of the user asks during Kubeflow Summit at KubeCon + CloudNativeCon 2025 NA.

cc @kubeflow/kubeflow-sdk-team @kubeflow/kubeflow-trainer-team @Fiona-Waters @austinlparker @alolita

### Why is this needed?

Improve observability experience for Kubeflow SDK users.

### Love this feature?

Give it a 👍 We prioritize the features with most 👍