kubeflow / kubeflow/sdk

bug(trainer): LocalJob.logs(follow=True) prints directly to stdout instead of yielding to caller

Open
#435 2 comments 0 reactions 0 assignees View on GitHub
area/local kind/bug
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What happened?

Two bugs in `LocalJob.logs(follow=True)` that make it pretty unusable as a streaming interface:

**1. Duplicate output.** The method calls `print()` internally, so if you iterate `get_job_logs(follow=True)` and print each line yourself, everything shows up twice — once from inside the SDK, once from your own code.

**2. No real-time streaming.** Despite `follow=True` implying otherwise, nothing is yielded until the job finishes. It's just a batched return wearing a streaming API's clothes.

### Steps to Reproduce

```python
from kubeflow.trainer import TrainerClient
from kubeflow.trainer.backends.localprocess.backend import LocalProcessBackendConfig

client = TrainerClient(backend_config=LocalProcessBackendConfig())
job_id = client.train(...)

for line in client.get_job_logs(name=job_id, follow=True):
print(line) # every line prints twice
```

### Where it's happening

`kubeflow/trainer/backends/localprocess/job.py` — `logs()` method:

```python
def logs(self, follow=False) -> list[str]:
if not follow:
return self._stdout.splitlines()
try:
for chunk in self.stream_logs():
print(chunk, end="", flush=True) # ← writes directly to stdout, caller has no say
except StopIteration:
pass
return self._stdout.splitlines() # ← blocks until job is done, then dumps everything at once
```

### What should happen instead

- No `print()` inside the method — yielding lines and leaving output decisions to the caller is the whole point of this API.
- Lines should reach the caller as they're produced, not in one batch after the job exits.

### Environment

- Kubeflow SDK: `0.4.0`
- Backend: `LocalProcessBackend`
- No Kubernetes cluster needed to reproduce

/kind bug
/area local

Contributor guide

Open the contributing guide

Research direction

Start in kubeflow/trainer/backends/localprocess/job.py at LocalJob.logs() and trace the stream_logs() path. Reproduce the get_job_logs(follow=True) example, then verify that output reaches the caller incrementally without being printed internally and that non-following logs retain their existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.