GoogleCloudPlatform / GoogleCloudPlatform/DataflowTemplates

[Bug]: VECTOR LENGTH not supported in Spanner to BigQuery Change Stream Processor

Open
#4,120 2 comments 0 reactions 0 assignees View on GitHub
bug p2
Dominant language
Java
Stars
1.3k
Forks
1.1k
Avg merge
2d 20h
Merged PRs (30d)
60

Description

### Related Template(s)

spanner_change_streams_to_bigquery

### Template Version

2026-05-26-00_RC00

### What happened?

I have a spanner enterprise table with a column

```sql
CREATE TABLE IF NOT EXISTS user_face_embeddings_buffalo_l (
id VARCHAR(36) NOT NULL,
embedding float4[] VECTOR LENGTH 512 NOT NULL -- this column caused problems
)
```

and I have the changestream to bigquery job deployed. Right when I landed this table with no data, the dataflow job started printing the following errors

```
Error message from worker: generic::unknown: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: java.lang.RuntimeException: Caught exception when constructing SpannerColumn for each column tracked by change stream spanner_snapshot_stream, message: Optional[Unsupported Spanner PostgreSQL type: real[] vector length 512], cause: null
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.FailsafeModJsonToTableRowTransformer$FailsafeModJsonToTableRow$FailsafeModJsonToTableRowFn$DoFnInvoker$6e2a6220.invokeSetup(Unknown Source)
org.apache.beam.sdk.transforms.reflect.DoFnInvokers.tryInvokeSetupFor(DoFnInvokers.java:53)
org.apache.beam.fn.harness.FnApiDoFnRunner.(FnApiDoFnRunner.java:442)
org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.addRunnerForPTransform(FnApiDoFnRunner.java:181)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:304)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:876)
org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:510)
org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:988)
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:506)
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:150)
org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:115)
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run(UnboundedScheduledExecutorService.java:163)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Caught exception when constructing SpannerColumn for each column tracked by change stream spanner_snapshot_stream, message: Optional[Unsupported Spanner PostgreSQL type: real[] vector length 512], cause: null
com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.FailsafeModJsonToTableRowTransformer$FailsafeModJsonToTableRow$FailsafeModJsonToTableRowFn.setUp(FailsafeModJsonToTableRowTransformer.java:197)
Caused by: java.lang.RuntimeException: Caught exception when constructing SpannerColumn for each column tracked by change stream spanner_snapshot_stream, message: Optional[Unsupported Spanner PostgreSQL type: real[] vector length 512], cause: null
com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.schemautils.SpannerChangeStreamsUtils.getSpannerColumnsByTableName(SpannerChangeStreamsUtils.java:234)
com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.schemautils.SpannerChangeStreamsUtils.getSpannerTableByName(SpannerChangeStreamsUtils.java:119)
com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.schemautils.SpannerChangeStreamsUtils.getSpannerTableByName(SpannerChangeStreamsUtils.java:105)
com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.FailsafeModJsonToTableRowTransformer$FailsafeModJsonToTableRow$FailsafeModJsonToTableRowFn.setUp(FailsafeModJsonToTableRowTransformer.java:190)
com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.FailsafeModJsonToTableRowTransformer$FailsafeModJsonToTableRow$FailsafeModJsonToTableRowFn$DoFnInvoker$6e2a6220.invokeSetup(Unknown Source)
org.apache.beam.sdk.transforms.reflect.DoFnInvokers.tryInvokeSetupFor(DoFnInvokers.java:53)
org.apache.beam.fn.harness.FnApiDoFnRunner.(FnApiDoFnRunner.java:442)
org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.addRunnerForPTransform(FnApiDoFnRunner.java:181)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:304)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:876)
org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:510)
org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:988)
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:506)
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:150)
org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:115)
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run(UnboundedScheduledExecutorService.java:163)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.IllegalArgumentException: Unsupported Spanner PostgreSQL type: real[] vector length 512
com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.schemautils.TypesUtils.informationSchemaPostgreSQLTypeToSpannerType(TypesUtils.java:98)
com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.schemautils.SpannerChangeStreamsUtils.informationSchemaTypeToSpannerType(SpannerChangeStreamsUtils.java:489)
com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.schemautils.SpannerChangeStreamsUtils.getSpannerColumnsByTableName(SpannerChangeStreamsUtils.java:222)
... 26 more

passed through:
==>
dist_proc/dax/workflow/worker/fnapi_service_impl.cc:1347
```

### Relevant log output

```shell

```

Contributor guide

Open the contributing guide

Research direction

The failure originates in TypesUtils.java at informationSchemaPostgreSQLTypeToSpannerType, called from SpannerChangeStreamsUtils.java while mapping change-stream schema metadata. Start there and trace how PostgreSQL array types are handled. Done means a real[] VECTOR LENGTH 512 column is accepted by the Spanner-to-BigQuery change stream processor without the reported unsupported-type exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.