DatastoreV1.write() Should Optionally Return PCollection<Void>
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
I'm trying to write to Datastore, wait for it to complete, then perform a task. In order to do this, the DatastoreV1.write() transform would need to return PCollection. Looking at the code, it returns PDone, and I can't find a workaround.
I propose adding an option to return PCollection. If I create a chained method like this then I think I can avoid breaking backwards compatibility:
```
DatastoreIO.v1().write().withReturn()
```
Alternatively, an entirely new write() method:
```
DatastoreIO.v1().writeWithReturn()
```
To illustrate the use case:
```
PCollection collection = someMethodToReturnPCollection();
PCollection datastoreWrites
= collection
.apply(
DatastoreIO.v1().write().withProjectId(projectId)
);
PCollection
otherCollection = ... // get the collection
otherCollection
.apply(
Wait.on(datastoreWrites)
)
.apply(
SomeOtherTransform()
);
```
I'll be creating a pull request soon to implement this.
Imported from Jira [BEAM-9491](https://issues.apache.org/jira/browse/BEAM-9491). Original Jira may contain additional context.
Reported by: alec.ferguson.
Contributor guide
Assessment
This issue has not been assessed yet.