GoogleCloudPlatform / GoogleCloudPlatform/DataflowTemplates
[Bug]: sourcedb to spanner does not accept spanner.tokenlist virtual columns
- Dominant language
- Java
- Stars
- 1.3k
- Forks
- 1.1k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 60
Description
### Related Template(s)
sourcedb_to_spanner_flex
### Template Version
2026-08-25-00_rc00
### What happened?
My Spanner (target) DDL contains this:
```sql
CREATE TABLE foo (
id bigint GENERATED BY DEFAULT AS IDENTITY NOT NULL,
name character varying NOT NULL,
name_tokens spanner.tokenlist GENERATED ALWAYS AS (spanner.tokenize_substring(name)) VIRTUAL HIDDEN,
PRIMARY KEY(id)
);
```
And it seems there are some validation of the Spanner supported column types, and it seems `spanner.tokenlist` is not accepted. But, because the column is generated, IMHO, it should be ignored as this column will be built by Spanner directly, and not by the dataflow pipeline.
### Relevant log output
```shell
com.google.cloud.teleport.v2.common.UncaughtExceptionLogger - The template launch failed.
java.lang.IllegalArgumentException: Unknown spanner type spanner.tokenlist
at com.google.cloud.teleport.v2.spanner.ddl.Column.parseSpannerType(Column.java:469)
at com.google.cloud.teleport.v2.spanner.ddl.Column$Builder.parseType(Column.java:326)
at com.google.cloud.teleport.v2.spanner.ddl.InformationSchemaScanner.listColumns(InformationSchemaScanner.java:200)
at com.google.cloud.teleport.v2.spanner.ddl.InformationSchemaScanner.scan(InformationSchemaScanner.java:62)
at com.google.cloud.teleport.v2.spanner.migrations.spanner.SpannerSchema.getInformationSchemaAsDdl(SpannerSchema.java:53)
at com.google.cloud.teleport.v2.templates.PipelineController.executeMigrationForDbConfigContainer(PipelineController.java:65)
at com.google.cloud.teleport.v2.source.jdbc.AbstractJdbcSrcToSpSourceConnector.executeMigration(AbstractJdbcSrcToSpSourceConnector.java:73)
at com.google.cloud.teleport.v2.templates.SourceDbToSpanner.run(SourceDbToSpanner.java:145)
at com.google.cloud.teleport.v2.templates.SourceDbToSpanner.main(SourceDbToSpanner.java:83)
```
Contributor guide
Research direction
Start in com.google.cloud.teleport.v2.spanner.ddl.Column.parseSpannerType and follow InformationSchemaScanner.listColumns, using the stack trace as the entry path. Check how generated columns are represented in the scanned schema, then verify that a sourcedb_to_spanner_flex migration accepts the spanner.tokenlist virtual column without failing on its type.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100