GoogleCloudPlatform / GoogleCloudPlatform/DataflowTemplates

[Bug]: sourcedb to spanner does not accept spanner.tokenlist virtual columns

Open
#4,204 0 comments 0 reactions 0 assignees View on GitHub
bug needs triage p2
Dominant language
Java
Stars
1.3k
Forks
1.1k
Avg merge
2d 20h
Merged PRs (30d)
60

Description

### Related Template(s)

sourcedb_to_spanner_flex

### Template Version

2026-08-25-00_rc00

### What happened?

My Spanner (target) DDL contains this:

```sql
CREATE TABLE foo (
id bigint GENERATED BY DEFAULT AS IDENTITY NOT NULL,
name character varying NOT NULL,
name_tokens spanner.tokenlist GENERATED ALWAYS AS (spanner.tokenize_substring(name)) VIRTUAL HIDDEN,
PRIMARY KEY(id)
);
```

And it seems there are some validation of the Spanner supported column types, and it seems `spanner.tokenlist` is not accepted. But, because the column is generated, IMHO, it should be ignored as this column will be built by Spanner directly, and not by the dataflow pipeline.

### Relevant log output

```shell
com.google.cloud.teleport.v2.common.UncaughtExceptionLogger - The template launch failed.
java.lang.IllegalArgumentException: Unknown spanner type spanner.tokenlist
at com.google.cloud.teleport.v2.spanner.ddl.Column.parseSpannerType(Column.java:469)
at com.google.cloud.teleport.v2.spanner.ddl.Column$Builder.parseType(Column.java:326)
at com.google.cloud.teleport.v2.spanner.ddl.InformationSchemaScanner.listColumns(InformationSchemaScanner.java:200)
at com.google.cloud.teleport.v2.spanner.ddl.InformationSchemaScanner.scan(InformationSchemaScanner.java:62)
at com.google.cloud.teleport.v2.spanner.migrations.spanner.SpannerSchema.getInformationSchemaAsDdl(SpannerSchema.java:53)
at com.google.cloud.teleport.v2.templates.PipelineController.executeMigrationForDbConfigContainer(PipelineController.java:65)
at com.google.cloud.teleport.v2.source.jdbc.AbstractJdbcSrcToSpSourceConnector.executeMigration(AbstractJdbcSrcToSpSourceConnector.java:73)
at com.google.cloud.teleport.v2.templates.SourceDbToSpanner.run(SourceDbToSpanner.java:145)
at com.google.cloud.teleport.v2.templates.SourceDbToSpanner.main(SourceDbToSpanner.java:83)
```

Contributor guide

Open the contributing guide

Research direction

Start in com.google.cloud.teleport.v2.spanner.ddl.Column.parseSpannerType and follow InformationSchemaScanner.listColumns, using the stack trace as the entry path. Check how generated columns are represented in the scanned schema, then verify that a sourcedb_to_spanner_flex migration accepts the spanner.tokenlist virtual column without failing on its type.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.