Data-dependent sinks
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Current sink API writes all data to a single destination, but there are many use cases where different pieces of data need to be routed to different destinations where the set of destinations is data-dependent (so can't be implemented with a Partition transform).
One internally discussed proposal was an API of the form:
```
PCollection PCollection.apply(
Write.using(DoFn where,
MapFn> how)
```
so an item T gets written to a destination (or multiple destinations) determined by "where"; and the writing strategy is determined by "how" that produces a WriteOperation (current API - global init/write/global finalize hooks) for any given destination.
This API also has other benefits:
* allows the SinkT to be computed dynamically (in "where"), rather than specified at pipeline construction time
* removes the necessity for a Sink class entirely
* is sequenceable w.r.t. downstream transforms (you can stick transforms onto the returned PCollection, while the current Write.to() returns a PDone)
Imported from Jira [BEAM-92](https://issues.apache.org/jira/browse/BEAM-92). Original Jira may contain additional context.
Reported by: jkff.
Contributor guide
Research direction
Start by reading the current sink API and Write.to() behavior described in the issue, then review the original Jira BEAM-92 for additional context. Define and implement the proposed data-dependent Write API so destinations can be computed per item and the returned PCollection can be used by downstream transforms; identify the relevant tests while tracing the existing sink implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100