Improve SpannerIO output
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
from a discussion in [https://github.com/apache/beam/pull/8097](https://github.com/apache/beam/pull/8097)
SpannerIO produces 2 output PCollections:
* getOutput() -\> PCollection
** never has any values
** in GlobalWindow
** Closed when the input PCollection is closed (ie never in streaming) to indicate when all input has been written
** Used in batch pipelines to have 'dependant' bulk imports - where one dataset is not written to Spanner until another has completed writing. (necessary for handling parent/child (1-many) referential integrity)
* getFailedMutations() -\> PCollection
** only contains values when Mutation[Group]s fail to be written
** in GlobalWindow
** Not very useful, as the reason for the failure is not given.
Suggestion:
* Deprecate these existing outputs.
* Make primary output be a PCollection<\{ MutationGroup, CommitTimestamp }\> so that the successfully written Mutation[Groups] can be processed further if necessary.
(\{a,b} signifies a container class for these values)
* Add an additional output of failed mutations PCollection<\{ MutationGroup, FailureMessage}\>
* The existing outputs can be derived from these new outputs
This allows useful error reporting/handling from the failure message, and the ability to continue processing the successful mutations.
(see also BEAM-6887)
Imported from Jira [BEAM-6921](https://issues.apache.org/jira/browse/BEAM-6921). Original Jira may contain additional context.
Reported by: nielm.
Contributor guide
Research direction
Start with SpannerIO and the discussion in PR 8097, then review BEAM-6887 for related context. Define the output design for successful and failed MutationGroups, including commit timestamps and failure messages, while preserving the existing outputs through the new outputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100