kubeflow / kubeflow/sdk

feat(common): add shared telemetry module for optional OpenTelemetry instrumentation

Open
#399 1 comment 0 reactions 0 assignees View on GitHub
kind/feature needs-triage
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What you would like to be added?

## Context

Following the discussion in #164, this issue tracks the implementation
of the shared foundation module for OpenTelemetry instrumentation across
all Kubeflow SDK clients.

## Proposed Location

`kubeflow/common/telemetry.py`

## Design

- `get_tracer(name)` — returns real OTel tracer or zero-overhead NoOp
- `SpanNames` — canonical span name constants for all SDK clients
- `SpanAttributes` — canonical attribute key constants
- `configure(exporter, endpoint)` — user-facing one-call setup
- `_NoOpTracer` / `_NoOpSpan` — zero overhead when opentelemetry-api
not installed

## Key Principles

- `opentelemetry-api` is optional — not in core dependencies
- Zero overhead when not installed — NoOp with empty method bodies
- Polling loops: per-iteration child spans with span events
- Single source of truth for naming across all SDK clients

## Span Hierarchy
kubeflow.sdk.trainer.train
├── kubeflow.sdk.trainer.get_runtime
├── kubeflow.sdk.trainer.create_trainjob
└── kubeflow.sdk.trainer.poll_status [× N iterations]
events: status_check, job_reached_expected_status

Related to #164

### Why is this needed?

The Kubeflow SDK currently has no observability into its internal operations. When `TrainerClient.train()` runs, users cannot see runtime resolution, CRD submission, or polling behavior. Debugging failures and measuring latency requires digging into raw Kubernetes events.

A shared `telemetry.py` module in `kubeflow/common/` provides the foundation for all SDK clients to emit OTel traces without duplicating instrumentation code, and without any overhead for users who don't enable tracing.

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Research direction

Start by reading the related discussion in #164 and the proposed kubeflow/common/telemetry.py location. Define the shared tracer, span-name and attribute constants, configuration entry point, and no-op behavior described in the issue; done means optional OpenTelemetry support works without core dependencies or overhead when disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.