GoogleCloudPlatform / GoogleCloudPlatform/DataflowTemplates

[Feature Request]: Datastream to Spanner Dataflow template to support Spanner datastream type

Open
#4,074 3 comments 0 reactions 0 assignees View on GitHub
addition p2
Dominant language
Java
Stars
1.3k
Forks
1.1k
Avg merge
2d 20h
Merged PRs (30d)
60

Description

### Related Template(s)

Primary: Datastream to Cloud Spanner template
Secondary: Spanner to SourceDB template, SourceDB to Spanner template

### What feature(s) are you requesting?

Currently, you can build a Datastream to stream Spanner CDC into GCS bucket: https://docs.cloud.google.com/datastream/docs/sources-spanner
There also exist a complementary Dataflow template to consume Datastream from GCS and stream to Spanner: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/datastream-to-spanner/README_Cloud_Datastream_to_Spanner.md

Unfortunately, the two are not compatible with each other since the Dataflow template does not allow Spanner as a Datastream source type, and other types are not compatible with Spanner: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/datastream-to-spanner/src/main/java/com/google/cloud/teleport/v2/templates/source/DatastreamToSpannerSourceConnectorRegistry.java#L35-L50

The request is to add support for Spanner within the template by implementing a complementary connector, so that it is possible to build Spanner-Spanner live migrations as per: https://docs.cloud.google.com/spanner/docs/data-migration#live-data-migration

Alternatively, if this route is not planned to be supported, or if there is another, supported way to perform live migrations between Spanner databases, the request is to cover that route in documentation and tooling. So far, I have reviewed multiple possible routes and some seem possible but not covered, so it is not clear whether they are supported at all.

### Workarounds

Performing "Spanner instance move" **is not the same use case**. It moves entire Spanner instances as-is, which is not suitable for cases like re-architecting the Spanner instances, which is what we currently do. As such, we need a way to perform a live migration of one Spanner DB at a time, between 2 arbitrary Spanner instances.

The workaround I found and tried is that you _can_ stream your CDC directly between Spanner instances using Spanner to SourceDB streaming template, however this seems to lack official support and is not documented, so I have low confidence in this solution: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/spanner-to-sourcedb/README.md / https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/spanner-to-sourcedb/src/main/java/com/google/cloud/teleport/v2/templates/SpannerToSourceDb.java#L410

Yet another workaround, which I haven't tried, is to try the other template (Source DB to Spanner), which seems to accept just about any JDBC driver and connection string, so in theory you can provide Spanner JDBC driver. However, it lacks documentation on which drivers are available OOTB and which needs to be supplied by the user: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/sourcedb-to-spanner/README.md#optional-parameters

Thank you!

Contributor guide

Open the contributing guide

Research direction

Start with v2/datastream-to-spanner/src/main/java/com/google/cloud/teleport/v2/templates/source/DatastreamToSpannerSourceConnectorRegistry.java and its README. Compare the referenced spanner-to-sourcedb and sourcedb-to-spanner templates and documentation to establish which migration route is supported. Done means either a supported Spanner source connector with documented live-migration behavior, or clear documentation and tooling for the supported alternative.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud, data-engineering, databases, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.