apache / apache/beam

Add "Write.withResults()" for all Java SDK IOs

Open
#21,329 1 comment 0 reactions 0 assignees View on GitHub
ideas improvement io P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Currently, by default, a "Write" transform is considered as a final `PTransform` in a pipeline and it returns `PDone` that can't be used later for next pipeline stages.

Though, sometimes it's needed to use the results of write operation as input `PCollection` downstream of pipeline and some IOs already support this. For example, `JdbcIO.Write` has the methods `withResults()` and `withWriteResults()` that returns a `PTransform` with output `PCollection<>` of write results and it can be used to analyse the results or, along with `Wait` transform, to write into the second database only if writes into the first one were finished, like this:

```

PCollection firstWriteResults = data.apply(JdbcIO.write()
.withDataSourceConfiguration(CONF_DB_1).withResults());
data.apply(Wait.on(firstWriteResults))

.apply(JdbcIO.write().withDataSourceConfiguration(CONF_DB_2));

```

So, it would be useful to add a `WriteWithResults` transform for other IOs that can be used in the same way.

Imported from Jira [BEAM-13584](https://issues.apache.org/jira/browse/BEAM-13584). Original Jira may contain additional context.
Reported by: aromanenko.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.