apache / apache/beam

Spanner ReaderWrite Transactions

Open
#20,493 1 comment 1 reaction 0 assignees View on GitHub
gcp io java P3 wish
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

We are currently using SpannerIO Mutations for writing Records to SpannerDB using both streaming and batch pipelines: 

However, our business requirements need more functionality in spannerio to be able to handle readwrite transactions. Some of the problems we are facing are:

*Scenario 1:* We need to make sure that the data that gets processed in the pipeline and committed into Spanner to be always the latest data. Hence we need the capability to be able to read / query specific field eg Date or SequenceNumber to compare with before committing the updated record into Spanner. Eg. only commit a certain record when the current event’s sequenceNumber is greater than whats already in the Spanner row.

*Scenario 2:* We need to run end of the day batch pipelines which is reconcile any missing data from streaming pipelines, but however if there is some latest data from streaming during the time of batch data write, we are unable to identify which is the latest row as they are two different pipelines

*Scenario 3:* In a streaming pipeline if an event was delayed and processed at later time by adding to backlog queue, we would not want to perform a Spanner update query on this event where there is relatively new event to the spanner db already.

If SpannerIO has some capability to create readwrite transactions. It would help us to extend it to be utilised in these kind of solutions. 

Could you please help us extend spannerio for such scenarios

Imported from Jira [BEAM-10489](https://issues.apache.org/jira/browse/BEAM-10489). Original Jira may contain additional context.
Reported by: sai.maduri.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.