GoogleCloudPlatform / GoogleCloudPlatform/DataflowTemplates
[Bug]: Improve the Datastream IT time
- Dominant language
- Java
- Stars
- 1.3k
- Forks
- 1.1k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 60
Description
### Related Template(s)
*datastream*
### Template Version
latest
### What happened?
[job-logs.txt](https://github.com/user-attachments/files/20738469/job-logs.txt)
From the previous release log, the Datastream ITs ran for 1h20m. We should optimize the ITs to reduce this time:
```
Time(s) Test_Name
1,704.025 com.google.cloud.teleport.v2.templates.MySQLSourceDbToSpannerSimpleIT
1,432.843 com.google.cloud.teleport.v2.templates.SpannerToSourceDbWideRowBasicIT
1,326.577 com.google.cloud.teleport.v2.templates.PostgreSQLDataTypesIT
1,278.379 com.google.cloud.teleport.v2.templates.endtoend.ForwardAndReverseMigrationEndToEndIT
1,276.618 com.google.cloud.teleport.v2.templates.MySQLDataTypesIT
1,101.849 com.google.cloud.teleport.v2.templates.DataStreamToSpannerWideRowForMaxTableNameWithMaxColumnNameIT
1,068.503 com.google.cloud.teleport.v2.templates.DataStreamToSpannerWideRowForMax16KeyTablePerDatabaseIT
1,059.708 com.google.cloud.teleport.v2.templates.DataStreamToSpannerWideRowForMaxColumnsPerTablesIT
1,054.155 com.google.cloud.teleport.v2.templates.SpannerToSourceDbTimezoneIT
1,034.398 com.google.cloud.teleport.v2.templates.DataStreamToSpannerWideRowForMax9MibTablePerDatabaseIT
996.763 com.google.cloud.teleport.v2.templates.GCSToSourceDbInterleaveMultiShardIT
994.383 com.google.cloud.teleport.v2.templates.DataStreamToSpannerWideRowForMaxTableNameIT
975.837 com.google.cloud.teleport.v2.templates.MySQLForeignKeyDependencyIT
957.755 com.google.cloud.teleport.v2.templates.GCSToSourceDbInterleaveIT
914.849 com.google.cloud.teleport.v2.templates.MySQLSourceDBToSpannerWideRowInterleaveDepthIT
914.376 com.google.cloud.teleport.v2.templates.SpannerChangeStreamToGcsMultiShardIT
913.715 com.google.cloud.teleport.v2.templates.SpannerToSourceDbInterleaveMultiShardIT
912.042 com.google.cloud.teleport.v2.templates.TimezoneIT
904.415 com.google.cloud.teleport.v2.templates.GCSToSourceDbWithReaderIT
883.866 com.google.cloud.teleport.v2.templates.SpannerToSourceDbIT
875.497 com.google.cloud.teleport.v2.templates.SpannerToCassandraSourceDbIT
866.845 com.google.cloud.teleport.v2.templates.MySQLIdentitySchemaMapperWithTransformationIT
844.141 com.google.cloud.teleport.v2.templates.DataStreamToSpannerShardedMigrationWithMigrationShardIdColumnIT
841.219 com.google.cloud.teleport.v2.templates.PostgreSQLWithUniformizationIT
835.774 com.google.cloud.teleport.v2.templates.CassandraAllDataTypesIT
829.831 com.google.cloud.teleport.v2.templates.SpannerToCassandraSourceDBCustomTransformationIT
824.275 com.google.cloud.teleport.v2.templates.MySQLIdentitySchemaMapperIT
811.819 com.google.cloud.teleport.v2.templates.MySQLStringOverridesSchemaMapperIT
807.204 com.google.cloud.teleport.v2.templates.MySQLFileOverridesSchemaMapperIT
806.921 com.google.cloud.teleport.v2.templates.MySQLSessionSchemaMapperWithTransformationIT
806.897 com.google.cloud.teleport.v2.templates.MySQLSourceDBToSpannerWideRowMaxSizeStringIT
798.366 com.google.cloud.teleport.v2.templates.MySQLDDLIT
797.364 com.google.cloud.teleport.v2.templates.MySQLSessionSchemaMapperWithTableFilterIT
795.596 com.google.cloud.teleport.v2.templates.MySQLSessionSchemaMapperIT
795.433 com.google.cloud.teleport.v2.templates.MySQLSingleShardIT
794.264 com.google.cloud.teleport.v2.templates.MySQLToSpannerWiderowForMaxColumnsPerTableIT
791.747 com.google.cloud.teleport.v2.templates.PostgreSQLReservedKeywordsIT
788.259 com.google.cloud.teleport.v2.templates.MySQLSourceDBToSpannerWideRowMaxColumnAndTableSizeIT
783.82 com.google.cloud.teleport.v2.templates.MySQLSourceDbToSpannerWideRowMaxColumnsTableKeyIT
775.308 com.google.cloud.teleport.v2.templates.MySQLSourceDbToSpannerWideRowMaxSizeTableKeyIT
774.154 com.google.cloud.teleport.v2.templates.SpannerToSourceDbCustomShardIT
773.361 com.google.cloud.teleport.v2.templates.SpannerToMySqlSourceDbWideRowMaxColumnsIT
770.478 com.google.cloud.teleport.v2.templates.SpannerToCassandraSourceDbMaxColumnsIT
766.912 com.google.cloud.teleport.v2.templates.PostgreSQLIdentitySchemaMapperIT
764.168 com.google.cloud.teleport.v2.templates.PostgreSQLNamespaceIT
748.35 com.google.cloud.teleport.v2.templates.MySQLCustomTransformationsNonShardedIT
736.117 com.google.cloud.teleport.v2.templates.SpannerToCassandraSourceDbMaxColumnsSizeIT
665.913 com.google.cloud.teleport.v2.templates.SpannerToSourceDbDatatypeIT
652.85 com.google.cloud.teleport.v2.templates.SpannerChangeStreamToGcsSingleShardIT
625.725 com.google.cloud.teleport.v2.templates.SpannerToCassandraSourceDbWideRow10MbIT
622.319 com.google.cloud.teleport.v2.templates.DataStreamToSpannerShardedMigrationWithoutMigrationShardIdColumnIT
616.5 com.google.cloud.teleport.v2.templates.SeparateShadowTableDatabaseShardedMigrationWithoutMigrationShardIdColumnIT
616.091 com.google.cloud.teleport.v2.templates.SpannerChangeStreamToGcsCustomShardIT
612.72 com.google.cloud.teleport.v2.templates.SpannerToMySqlSourceDbWideRow10MbIT
564.574 com.google.cloud.teleport.v2.templates.SeparateShadowTableDatabaseShardedMigrationWithMigrationShardIdColumnIT
538.91 com.google.cloud.teleport.v2.templates.SeparateShadowTableDatabaseEventsIT
522.797 com.google.cloud.teleport.v2.templates.GCSToSourceDbWithoutReaderIT
522.715 com.google.cloud.teleport.v2.templates.SeparateShadowTableDatabaseFileOverridesIT
496.858 com.google.cloud.teleport.v2.templates.SeparateShadowTableDatabaseMixedIT
493.773 com.google.cloud.teleport.v2.templates.DataStreamToSpannerMixedIT
488.191 com.google.cloud.teleport.v2.templates.SeparateShadowTableDatabasePKFocusedIT
471.673 com.google.cloud.teleport.v2.templates.DataStreamToSpannerSessionIT
463.04 com.google.cloud.teleport.v2.templates.DatastreamToSpannerSingleDFShardedMigrationIT
452.56 com.google.cloud.teleport.v2.templates.SeparateShadowTableDatabaseSessionIT
452.085 com.google.cloud.teleport.v2.templates.DataStreamToSpannerDDLIT
443.389 com.google.cloud.teleport.v2.templates.DataStreamToSpannerEventsIT
442.165 com.google.cloud.teleport.v2.templates.SeparateShadowTableDatabaseDDLIT
409.714 com.google.cloud.teleport.v2.templates.SeparateShadowTableDatabaseStringOverridesIT
399.11 com.google.cloud.teleport.v2.templates.SeparateShadowTableDatabaseSingleDFShardedMigrationIT
371.438 com.google.cloud.teleport.v2.templates.DataStreamToSpannerFileOverridesIT
368.936 com.google.cloud.teleport.v2.templates.DataStreamToSpannerStringOverridesIT
56.208 com.google.cloud.teleport.v2.templates.spanner.DatastreamToSpannerExceptionClassifierIT
```
### Relevant log output
```shell
```
Contributor guide
Assessment
This issue has not been assessed yet.