apache / apache/beam

WriteToFiles with a concrete sink can shunt data to a single window

Open
#20,849 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When given a concrete `FileSink`, `WriteToFiles` will re-use the same sink across windows:
- https://github.com/apache/beam/blob/e92d184abc79fe84c48de3dfd9dd168d9b38feac/sdks/python/apache_beam/io/fileio.py#L461
- https://github.com/apache/beam/blob/e92d184abc79fe84c48de3dfd9dd168d9b38feac/sdks/python/apache_beam/io/fileio.py#L625

This can lead to data for one window being written to the sink for another window.

See discussion: https://github.com/apache/beam/pull/14374#discussion_r604320333

Imported from Jira [BEAM-12072](https://issues.apache.org/jira/browse/BEAM-12072). Original Jira may contain additional context.
Reported by: bhulette.

Contributor guide

Open the contributing guide

Research direction

Start in sdks/python/apache_beam/io/fileio.py at the cited lines around WriteToFiles and the concrete FileSink handling, then read the linked discussion in PR 14374. Trace how sinks are reused across windows and verify that data for each window is written to its corresponding sink rather than a single window's sink.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.