GoogleCloudPlatform / GoogleCloudPlatform/DataflowTemplates

[Bug]: Spanner IT test suites are extremely flaky

Open
#4,254 0 comments 0 reactions 0 assignees View on GitHub
bug needs triage p2
Dominant language
Java
Stars
1.3k
Forks
1.1k
Avg merge
2d 20h
Merged PRs (30d)
60

Description

### Related Template(s)

Spanner

### Template Version

latest

### What happened?

See: https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/workflows/spanner-pr.yml?query=branch%3Amain

Most test runs fail.

Errors range from stockouts to test failures:

looked at 3 recent runs:

```

gha_triage https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34756800171 · 1 job(s) to inspect
· job 103724298431 (Spanner PR / Dataflow Templates Integrat): 106441 lines → 4 failure(s)
...
1. [unknown] 2 test(s)
suite Spanner PR / Dataflow Templates Integration Tests
cause ContainerLaunch: Container startup failed for image [mcr.microsoft.com/mssql/server:2025-latest](http://mcr.microsoft.com/mssql/server:2025-latest)
why no rule matched — needs a human
test SQLServerDataTypesIT.setUp (0s) · sourcedb-to-spanner
SQLServerDataTypesIT:65
test SQLServerDataTypesPGDialectIT.setUp (0s) · sourcedb-to-spanner
SQLServerDataTypesPGDialectIT:66
log https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34756800171/job/103724298431 (line ~106103)
── excerpt ──
| [ERROR] SQLServerDataTypesIT.setUp:65->SourceDbToSpannerITBase.setUpMSSQLResourceManager:164 » ContainerLaunch Container startup failed for image [mcr.microsoft.com/mssql/server:2025-latest](http://mcr.microsoft.com/mssql/server:2025-latest)

gha_triage https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34793738863
· 1 job(s) to inspect
· job 103825119090 (Spanner PR / Dataflow Templates Integrat): 79622 lines → 1 failure(s)

Integration test triage — run 34793738863
GoogleCloudPlatform/DataflowTemplates · 1 failing job(s) · triaged in 6.7s

Summary: 1 failing test(s) in 1 job(s), 1 distinct root cause(s)
1 likely real

Failing suites
• Spanner PR / Dataflow Templates Integration Tests 1 failure(s) LIKELY REAL BUG
main: ✗ failing (4h ago)
history: https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/workflows/spanner-pr.yml?query=branch%3Amain
analyze: python3 /google/src/head/depot/google3/experimental/users/valentyn/gha_triage/gha_triage.py https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34843186644

1. [LIKELY REAL BUG] 1 test(s)
suite Spanner PR / Dataflow Templates Integration Tests
cause expected that contains unordered record (and case insensitive) {ID=1, NAME=Parent Row 1}, but only had [{name=Child Row 1, id=2}, {name=Grandchild Row 1, id=3}]
why data assertion mismatch — pipeline output differs from expectation
test PostgreSQLDatastreamToSpannerInheritanceIT.testPostgreSqlInheritance (1181s) · spanner-to-sourcedb
com.google.cloud.teleport.v2.templates.PostgreSQLDatastreamToSpannerInheritanceIT.validateResult(PostgreSQLDatastreamToSpannerInheritanceIT.java:264)
log https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34793738863/job/103825119090 (line ~65809)
── excerpt ──
| expected that contains unordered record (and case insensitive) {ID=1, NAME=Parent Row 1}, but only had [{name=Child Row 1, id=2}, {name=Grandchild Row 1, id=3}]
| at com.google.cloud.teleport.v2.templates.PostgreSQLDatastreamToSpannerInheritanceIT.validateResult(PostgreSQLDatastreamToSpannerInheritanceIT.java:264)
| at com.google.cloud.teleport.v2.templates.PostgreSQLDatastreamToSpannerInheritanceIT.testPostgreSqlInheritance(PostgreSQLDatastreamToSpannerInheritanceIT.java:193)

⚠ Attention: at least one failure looks like a real defect.

$ gha_triage https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34843186644
· 1 job(s) to inspect
· job 103977588004 (Spanner PR / Dataflow Templates Integrat): 106186 lines → 1 failure(s)

Integration test triage — run 34843186644
GoogleCloudPlatform/DataflowTemplates · 1 failing job(s) · triaged in 8.4s

Summary: 1 failing test(s) in 1 job(s), 1 distinct root cause(s)
1 pipeline failed

...

1. [pipeline FAILED — check Dataflow job] 1 test(s)
suite Spanner PR / Dataflow Templates Integration Tests
cause RuntimeException: Job ID 2026-09-14_05_53_52-451419292641303686 under span-cloud-teleport-testing failed. Please check cloud console for more details.
why the launched Dataflow job failed — open the job below; this CI log only reports that it failed, not why
test BulkMigrationAndValidationMySQLAllDataTypesE2EIT.allDataTypesE2E (1237s) · spanner-to-sourcedb
org.apache.beam.it.common.PipelineOperator.jobIsDone(PipelineOperator.java:242)
df job https://console.cloud.google.com/dataflow/jobs/us-central1/2026-09-14_05_53_52-451419292641303686?project=span-cloud-teleport-testing
log https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34843186644/job/103977588004 (line ~63774)
── excerpt ──
| java.lang.RuntimeException: Job ID 2026-09-14_05_53_52-451419292641303686 under span-cloud-teleport-testing failed. Please check cloud console for more details.
| at org.apache.beam.it.common.PipelineOperator.jobIsDone(PipelineOperator.java:242)
| at org.apache.beam.it.common.PipelineOperator.lambda$waitUntilDone$1(PipelineOperator.java:67)
| at org.apache.beam.it.common.PipelineOperator.allMatch(PipelineOperator.java:284)
| at org.apache.beam.it.common.PipelineOperator.finishOrTimeout(PipelineOperator.java:211)
| at org.apache.beam.it.common.PipelineOperator.waitUntilDone(PipelineOperator.java:64)
| at com.google.cloud.teleport.v2.templates.endtoend.BulkMigrationAndValidationMySQLAllDataTypesE2EIT.allDataTypesE2E(BulkMigrationAndValidationMySQLAllDataTypesE2EIT.java:151)
| at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
| at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
| at java.base/java.lang.reflect.Method.invoke(Method.java:569)
| at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
| at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
| at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
| at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
| at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
| at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
| at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
| at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
| at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
| at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
| at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
| at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
| at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
| at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:345)
| … 5 more line(s) — open the log above

→ Pipeline reached FAILED state. Open the Dataflow job above; this log does not contain the cause.
```

The last seems to be stockout.

### Relevant log output

```shell

```

Contributor guide

Open the contributing guide

Research direction

Start with the spanner-pr.yml workflow and compare the three recent runs linked in the issue. Inspect SQLServerDataTypesIT, SQLServerDataTypesPGDialectIT, PostgreSQLDatastreamToSpannerInheritanceIT.validateResult, and BulkMigrationAndValidationMySQLAllDataTypesE2E for reproducible causes. Done means the Spanner integration suite completes reliably without these setup, assertion, or pipeline failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql, postgresql
Domain
ci-cd, databases, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.