Python SDK Fn harness should not log to stderr for portable jobs
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
We currently log both to stderr and the portable logging API, but we should do only the latter (and potentially fallback to stderr under failures). Otherwise, we're duplicating the log volume.
Location where the Fn Logger is added:
https://github.com/apache/beam/blob/f3dba1af43b03e24419fb0f5b726479368959eef/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L95
Imported from Jira [BEAM-3623](https://issues.apache.org/jira/browse/BEAM-3623). Original Jira may contain additional context.
Reported by: herohde.
Contributor guide
Research direction
Start at sdks/python/apache_beam/runners/worker/sdk_worker_main.py around line 95, where the Fn Logger is added, and trace how messages reach the portable logging API and stderr. Done means portable jobs no longer duplicate log messages on stderr, with the intended failure fallback behavior clarified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100