Add support for windowed filenames in Python SDK
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
The Python SDK's FilebasedSink does not support windowed filenames. Currently the only filename template available is based on shards.
The Java SDK has this feature in the `FilenamePolicy.windowedFilename()`. The python SDk only has a somewhat more simple implementation using a text template and `FileBasedSink._template_to_format()` which does not support windows.
Imported from Jira [BEAM-3070](https://issues.apache.org/jira/browse/BEAM-3070). Original Jira may contain additional context.
Reported by: while.
Contributor guide
Research direction
Start with the Python SDK's FileBasedSink and its _template_to_format() method, then compare the Java SDK's FilenamePolicy.windowedFilename() implementation. Use the BEAM-3070 context for additional requirements; done means Python filename templates support windowing rather than only shard-based naming.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100