WriteToFiles with a concrete sink can shunt data to a single window
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When given a concrete `FileSink`, `WriteToFiles` will re-use the same sink across windows:
- https://github.com/apache/beam/blob/e92d184abc79fe84c48de3dfd9dd168d9b38feac/sdks/python/apache_beam/io/fileio.py#L461
- https://github.com/apache/beam/blob/e92d184abc79fe84c48de3dfd9dd168d9b38feac/sdks/python/apache_beam/io/fileio.py#L625
This can lead to data for one window being written to the sink for another window.
See discussion: https://github.com/apache/beam/pull/14374#discussion_r604320333
Imported from Jira [BEAM-12072](https://issues.apache.org/jira/browse/BEAM-12072). Original Jira may contain additional context.
Reported by: bhulette.
Contributor guide
Research direction
Start in sdks/python/apache_beam/io/fileio.py at the cited lines around WriteToFiles and the concrete FileSink handling, then read the linked discussion in PR 14374. Trace how sinks are reused across windows and verify that data for each window is written to its corresponding sink rather than a single window's sink.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100