kubeflow / kubeflow/sdk

Spark log retrieval returns an extra empty line for empty or newline-terminated output

Open
#656 4 comments 0 reactions 1 assignee Claimed by @ADITYA-CODE-SOURCE View on GitHub
area/spark kind/bug
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What happened?

While reviewing the remaining Spark unit-test coverage in #609, I noticed that `read_pod_logs()` splits non-streaming log responses using `logs.split("\n")`.

This produces unexpected results:

```python
"".split("\n") == [""]
"line\n".split("\n") == ["line", ""]
"line 1\r\nline 2".split("\n") == ["line 1\r", "line 2"]
```

As a result, empty logs are represented as one blank line, newline-terminated logs contain an extra empty entry, and CRLF responses retain the carriage-return character.

`read_pod_logs()` is shared by both Spark Connect session logs and Spark batch-job logs.

Relevant code: `kubeflow/spark/backends/kubernetes/utils.py::read_pod_logs`

### What did you expect to happen?

Non-streaming log output should be returned as logical lines:

- Empty output should return no lines.
- A trailing newline should not create an additional empty entry.
- CRLF line endings should be handled correctly.
- Genuine blank lines inside the output should remain preserved.

Using `splitlines()` for the non-streaming response would provide this behavior. Unit tests should cover empty output, trailing newlines, CRLF output, and interior blank lines.

### Environment

Kubernetes version:

Not applicable. The issue is in client-side parsing of the string returned by the Kubernetes Python client and does not require a running cluster to reproduce.

Kubeflow Trainer version:

Not applicable.

Kubeflow Python SDK version:

Current `main` branch, observed at commit `8ecc659abbb13d089ee1046f5d967d5ad5a4da11`.

Relevant file: `kubeflow/spark/backends/kubernetes/utils.py`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.