apache / apache/beam

Improve SpannerIO output

Open
#19,404 0 comments 0 reactions 0 assignees View on GitHub
gcp io java new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

from a discussion in [https://github.com/apache/beam/pull/8097](https://github.com/apache/beam/pull/8097)
SpannerIO produces 2 output PCollections:
* getOutput() -\> PCollection
** never has any values
** in GlobalWindow
** Closed when the input PCollection is closed (ie never in streaming) to indicate when all input has been written
** Used in batch pipelines to have 'dependant' bulk imports - where one dataset is not written to Spanner until another has completed writing. (necessary for handling parent/child (1-many) referential integrity)
* getFailedMutations() -\> PCollection
** only contains values when Mutation[Group]s fail to be written
** in GlobalWindow
** Not very useful, as the reason for the failure is not given. 

Suggestion: 
* Deprecate these existing outputs.
* Make primary output be a PCollection<\{ MutationGroup, CommitTimestamp }\> so that the successfully written Mutation[Groups] can be processed further if necessary.
(\{a,b} signifies a container class for these values)
* Add an additional output of failed mutations PCollection<\{ MutationGroup, FailureMessage}\>
* The existing outputs can be derived from these new outputs

This allows useful error reporting/handling from the failure message, and the ability to continue processing the successful mutations. 

 

(see also BEAM-6887)

 

 

Imported from Jira [BEAM-6921](https://issues.apache.org/jira/browse/BEAM-6921). Original Jira may contain additional context.
Reported by: nielm.

Contributor guide

Open the contributing guide

Research direction

Start with SpannerIO and the discussion in PR 8097, then review BEAM-6887 for related context. Define the output design for successful and failed MutationGroups, including commit timestamps and failure messages, while preserving the existing outputs through the new outputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.