Add "Write.withResults()" for all Java SDK IOs
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Currently, by default, a "Write" transform is considered as a final `PTransform` in a pipeline and it returns `PDone` that can't be used later for next pipeline stages.
Though, sometimes it's needed to use the results of write operation as input `PCollection` downstream of pipeline and some IOs already support this. For example, `JdbcIO.Write` has the methods `withResults()` and `withWriteResults()` that returns a `PTransform` with output `PCollection<>` of write results and it can be used to analyse the results or, along with `Wait` transform, to write into the second database only if writes into the first one were finished, like this:
```
PCollection firstWriteResults = data.apply(JdbcIO.write()
.withDataSourceConfiguration(CONF_DB_1).withResults());
data.apply(Wait.on(firstWriteResults))
.apply(JdbcIO.write().withDataSourceConfiguration(CONF_DB_2));
```
So, it would be useful to add a `WriteWithResults` transform for other IOs that can be used in the same way.
Imported from Jira [BEAM-13584](https://issues.apache.org/jira/browse/BEAM-13584). Original Jira may contain additional context.
Reported by: aromanenko.
Contributor guide
Assessment
This issue has not been assessed yet.