apache / apache/beam

Fetch GCP-Datastore entity by id inside of a Dataflow transform

Open
#19,391 0 comments 0 reactions 0 assignees View on GitHub
ideas io new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I have a dataflow pipeline where first I queried one model, but then for each record that pointed to another model I wanted to be able to fetch that too.

Browsing the `apache_beam & google.cloud.proto.datastore.v1 `source I was able to make this happen:

[https://stackoverflow.com/a/55616668/4458510](https://stackoverflow.com/a/55616668/4458510)

I copied the same mechanism from the other Datastore DoFn's and it seems to work fine, and I think something like this would be a good addition to apache_beam.io.gcp.datastore.v1.datastoreio in the sdk

 

Imported from Jira [BEAM-7048](https://issues.apache.org/jira/browse/BEAM-7048). Original Jira may contain additional context.
Reported by: alexindaco.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.