SQL server connector on restart goes into OOM. [DBZ-2917]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 9
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-2917](https://issues.redhat.com/browse/DBZ-2917)
SQL server connector stops processing records when connection reset is received from the SQL server (This is being tracked under https://issues.redhat.com/browse/DBZ-2910). If we try to start the connector again, we run into OOM with following exception:
{noformat}
2021-01-12 04:04:41,657 - INFO [debezium-sqlserverconnector-dummy-change-event-source-coordinator:i.d.c.s.SqlServerStreamingChangeEventSource@113] - Last position recorded in offsets is 000000dd:000038c0:003b(000000dd:000038c0:003a)[0]
2021-01-12 04:09:09,689 - ERROR [debezium-sqlserverconnector-dummy-change-event-source-coordinator:i.d.p.ErrorHandler@31] - Producer failure
java.lang.OutOfMemoryError: Java heap space
at com.microsoft.sqlserver.jdbc.TDSPacket.(IOBuffer.java:6241) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6433) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at com.microsoft.sqlserver.jdbc.TDSCommand.detach(IOBuffer.java:7280) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:3032) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:247) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:222) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:444) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at io.debezium.jdbc.JdbcConnection.prepareQuery(JdbcConnection.java:596) ~[debezium-core-1.3.1.Final.jar:1.3.1.Final]
at io.debezium.connector.sqlserver.SqlServerConnection.getChangesForTables(SqlServerConnection.java:221) ~[1610424275119-0/:na]
at io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource.execute(SqlServerStreamingChangeEventSource.java:164) ~[1610424275119-0/:na]
at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:140) ~[debezium-core-1.3.1.Final.jar:1.3.1.Final]
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:113) ~[debezium-core-1.3.1.Final.jar:1.3.1.Final]
at io.debezium.pipeline.ChangeEventSourceCoordinator$$Lambda$374/1008041224.run(Unknown Source) ~[na:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_275]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
2021-01-12 04:09:09,690 - INFO [debezium-sqlserverconnector-dummy-change-event-source-coordinator:i.d.p.m.StreamingChangeEventSourceMetrics@60] - Connected metrics set to 'false' {noformat}
Tables we are replicating are 10, each having 27 columns. Amount of data in the DB is around 4GB and Java container running connector has heap size of 12 GB. Any connection configurations we need to tune? Debezium configurations we use are listed here - [https://github.com/data-integrations/database-delta-plugins/blob/develop/sqlserver-delta-plugins/src/main/java/io/cdap/delta/sqlserver/SqlServerEventReader.java#L102]
Contributor guide
Research direction
Start with sqlserver-delta-plugins/src/main/java/io/cdap/delta/sqlserver/SqlServerEventReader.java at the referenced configuration, then follow the stack trace through SqlServerConnection.getChangesForTables and SqlServerStreamingChangeEventSource.execute. Reproduce a connection reset followed by connector restart and inspect the query and connection configuration involved. Done means the connector restarts without exhausting the 12 GB heap and resumes processing records.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100