GoogleCloudPlatform / GoogleCloudPlatform/DataflowTemplates
[Bug]: Spanner IT test suites are extremely flaky
- Dominant language
- Java
- Stars
- 1.3k
- Forks
- 1.1k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 60
Description
### Related Template(s)
Spanner
### Template Version
latest
### What happened?
See: https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/workflows/spanner-pr.yml?query=branch%3Amain
Most test runs fail.
Errors range from stockouts to test failures:
looked at 3 recent runs:
```
gha_triage https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34756800171 · 1 job(s) to inspect
· job 103724298431 (Spanner PR / Dataflow Templates Integrat): 106441 lines → 4 failure(s)
...
1. [unknown] 2 test(s)
suite Spanner PR / Dataflow Templates Integration Tests
cause ContainerLaunch: Container startup failed for image [mcr.microsoft.com/mssql/server:2025-latest](http://mcr.microsoft.com/mssql/server:2025-latest)
why no rule matched — needs a human
test SQLServerDataTypesIT.setUp (0s) · sourcedb-to-spanner
SQLServerDataTypesIT:65
test SQLServerDataTypesPGDialectIT.setUp (0s) · sourcedb-to-spanner
SQLServerDataTypesPGDialectIT:66
log https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34756800171/job/103724298431 (line ~106103)
── excerpt ──
| [ERROR] SQLServerDataTypesIT.setUp:65->SourceDbToSpannerITBase.setUpMSSQLResourceManager:164 » ContainerLaunch Container startup failed for image [mcr.microsoft.com/mssql/server:2025-latest](http://mcr.microsoft.com/mssql/server:2025-latest)
gha_triage https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34793738863
· 1 job(s) to inspect
· job 103825119090 (Spanner PR / Dataflow Templates Integrat): 79622 lines → 1 failure(s)
Integration test triage — run 34793738863
GoogleCloudPlatform/DataflowTemplates · 1 failing job(s) · triaged in 6.7s
Summary: 1 failing test(s) in 1 job(s), 1 distinct root cause(s)
1 likely real
Failing suites
• Spanner PR / Dataflow Templates Integration Tests 1 failure(s) LIKELY REAL BUG
main: ✗ failing (4h ago)
history: https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/workflows/spanner-pr.yml?query=branch%3Amain
analyze: python3 /google/src/head/depot/google3/experimental/users/valentyn/gha_triage/gha_triage.py https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34843186644
1. [LIKELY REAL BUG] 1 test(s)
suite Spanner PR / Dataflow Templates Integration Tests
cause expected that contains unordered record (and case insensitive) {ID=1, NAME=Parent Row 1}, but only had [{name=Child Row 1, id=2}, {name=Grandchild Row 1, id=3}]
why data assertion mismatch — pipeline output differs from expectation
test PostgreSQLDatastreamToSpannerInheritanceIT.testPostgreSqlInheritance (1181s) · spanner-to-sourcedb
com.google.cloud.teleport.v2.templates.PostgreSQLDatastreamToSpannerInheritanceIT.validateResult(PostgreSQLDatastreamToSpannerInheritanceIT.java:264)
log https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34793738863/job/103825119090 (line ~65809)
── excerpt ──
| expected that contains unordered record (and case insensitive) {ID=1, NAME=Parent Row 1}, but only had [{name=Child Row 1, id=2}, {name=Grandchild Row 1, id=3}]
| at com.google.cloud.teleport.v2.templates.PostgreSQLDatastreamToSpannerInheritanceIT.validateResult(PostgreSQLDatastreamToSpannerInheritanceIT.java:264)
| at com.google.cloud.teleport.v2.templates.PostgreSQLDatastreamToSpannerInheritanceIT.testPostgreSqlInheritance(PostgreSQLDatastreamToSpannerInheritanceIT.java:193)
⚠ Attention: at least one failure looks like a real defect.
$ gha_triage https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34843186644
· 1 job(s) to inspect
· job 103977588004 (Spanner PR / Dataflow Templates Integrat): 106186 lines → 1 failure(s)
Integration test triage — run 34843186644
GoogleCloudPlatform/DataflowTemplates · 1 failing job(s) · triaged in 8.4s
Summary: 1 failing test(s) in 1 job(s), 1 distinct root cause(s)
1 pipeline failed
...
1. [pipeline FAILED — check Dataflow job] 1 test(s)
suite Spanner PR / Dataflow Templates Integration Tests
cause RuntimeException: Job ID 2026-09-14_05_53_52-451419292641303686 under span-cloud-teleport-testing failed. Please check cloud console for more details.
why the launched Dataflow job failed — open the job below; this CI log only reports that it failed, not why
test BulkMigrationAndValidationMySQLAllDataTypesE2EIT.allDataTypesE2E (1237s) · spanner-to-sourcedb
org.apache.beam.it.common.PipelineOperator.jobIsDone(PipelineOperator.java:242)
df job https://console.cloud.google.com/dataflow/jobs/us-central1/2026-09-14_05_53_52-451419292641303686?project=span-cloud-teleport-testing
log https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/34843186644/job/103977588004 (line ~63774)
── excerpt ──
| java.lang.RuntimeException: Job ID 2026-09-14_05_53_52-451419292641303686 under span-cloud-teleport-testing failed. Please check cloud console for more details.
| at org.apache.beam.it.common.PipelineOperator.jobIsDone(PipelineOperator.java:242)
| at org.apache.beam.it.common.PipelineOperator.lambda$waitUntilDone$1(PipelineOperator.java:67)
| at org.apache.beam.it.common.PipelineOperator.allMatch(PipelineOperator.java:284)
| at org.apache.beam.it.common.PipelineOperator.finishOrTimeout(PipelineOperator.java:211)
| at org.apache.beam.it.common.PipelineOperator.waitUntilDone(PipelineOperator.java:64)
| at com.google.cloud.teleport.v2.templates.endtoend.BulkMigrationAndValidationMySQLAllDataTypesE2EIT.allDataTypesE2E(BulkMigrationAndValidationMySQLAllDataTypesE2EIT.java:151)
| at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
| at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
| at java.base/java.lang.reflect.Method.invoke(Method.java:569)
| at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
| at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
| at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
| at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
| at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
| at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
| at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
| at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
| at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
| at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
| at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
| at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
| at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
| at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:345)
| … 5 more line(s) — open the log above
→ Pipeline reached FAILED state. Open the Dataflow job above; this log does not contain the cause.
```
The last seems to be stockout.
### Relevant log output
```shell
```
Contributor guide
Research direction
Start with the spanner-pr.yml workflow and compare the three recent runs linked in the issue. Inspect SQLServerDataTypesIT, SQLServerDataTypesPGDialectIT, PostgreSQLDatastreamToSpannerInheritanceIT.validateResult, and BulkMigrationAndValidationMySQLAllDataTypesE2E for reproducible causes. Done means the Spanner integration suite completes reliably without these setup, assertion, or pipeline failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql, postgresql
- Domain
- ci-cd, databases, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100