kubeflow / kubeflow/sdk

feat(trainer): Add log streaming and tailing support to TrainerClient.get_job_logs()

Open
#682 4 comments 0 reactions 2 assignees Claimed by @Sumitha007 View on GitHub
kind/feature needs-triage
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What you would like to be added?

Add parameter options for streaming logs line-by-line and limiting returned log lines to TrainerClient.get_job_logs().
Specifically:
1. follow: bool = False: When set to True, returns an Iterator[str] / generator that streams job logs in real time rather than returning a static list.
2. tail_lines: Optional[int] = None: Allows fetching only the last $N$ lines of log output.

Proposed Function Signature Update:

def get_job_logs(
self,
name: str,
follow: bool = False,
tail_lines: Optional[int] = None,
container_name: Optional[str] = None
) -> Union[List[str], Iterator[str]]:
"""Fetches or streams training job logs."""
...

### Why is this needed?

Currently, TrainerClient.get_job_logs() fetches all logs into memory at once as a single List[str]. This creates two main issues for long-running PyTorch, JAX, or Spark jobs:

1. Memory Exhaustion (OOM): Long training runs generate hundreds of megabytes or gigabytes of logs, causing client-side memory spikes and MemoryError crashes in interactive environments like Jupyter Notebooks.

2. Lack of Live Monitoring: Blocking execution until all logs are fetched prevents users from monitoring training loss and progress in real time as the job executes.

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.