debezium / debezium/dbz

SQL server connector on restart goes into OOM. [DBZ-2917]

Open
#407 0 comments 0 reactions 0 assignees View on GitHub
component/sqlserver-connector migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
9
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-2917](https://issues.redhat.com/browse/DBZ-2917)

SQL server connector stops processing records when connection reset is received from the SQL server (This is being tracked under https://issues.redhat.com/browse/DBZ-2910). If we try to start the connector again, we run into OOM with following exception:
{noformat}
2021-01-12 04:04:41,657 - INFO [debezium-sqlserverconnector-dummy-change-event-source-coordinator:i.d.c.s.SqlServerStreamingChangeEventSource@113] - Last position recorded in offsets is 000000dd:000038c0:003b(000000dd:000038c0:003a)[0]
2021-01-12 04:09:09,689 - ERROR [debezium-sqlserverconnector-dummy-change-event-source-coordinator:i.d.p.ErrorHandler@31] - Producer failure
java.lang.OutOfMemoryError: Java heap space
at com.microsoft.sqlserver.jdbc.TDSPacket.(IOBuffer.java:6241) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6433) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at com.microsoft.sqlserver.jdbc.TDSCommand.detach(IOBuffer.java:7280) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:3032) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:247) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:222) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:444) ~[mssql-jdbc-6.2.1.jre7.jar:na]
at io.debezium.jdbc.JdbcConnection.prepareQuery(JdbcConnection.java:596) ~[debezium-core-1.3.1.Final.jar:1.3.1.Final]
at io.debezium.connector.sqlserver.SqlServerConnection.getChangesForTables(SqlServerConnection.java:221) ~[1610424275119-0/:na]
at io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource.execute(SqlServerStreamingChangeEventSource.java:164) ~[1610424275119-0/:na]
at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:140) ~[debezium-core-1.3.1.Final.jar:1.3.1.Final]
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:113) ~[debezium-core-1.3.1.Final.jar:1.3.1.Final]
at io.debezium.pipeline.ChangeEventSourceCoordinator$$Lambda$374/1008041224.run(Unknown Source) ~[na:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_275]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_275]
2021-01-12 04:09:09,690 - INFO [debezium-sqlserverconnector-dummy-change-event-source-coordinator:i.d.p.m.StreamingChangeEventSourceMetrics@60] - Connected metrics set to 'false' {noformat}
 

Tables we are replicating are 10, each having 27 columns. Amount of data in the DB is around 4GB and Java container running connector has heap size of 12 GB. Any connection configurations we need to tune? Debezium configurations we use are listed here - [https://github.com/data-integrations/database-delta-plugins/blob/develop/sqlserver-delta-plugins/src/main/java/io/cdap/delta/sqlserver/SqlServerEventReader.java#L102]

Contributor guide

Open the contributing guide

Research direction

Start with sqlserver-delta-plugins/src/main/java/io/cdap/delta/sqlserver/SqlServerEventReader.java at the referenced configuration, then follow the stack trace through SqlServerConnection.getChangesForTables and SqlServerStreamingChangeEventSource.execute. Reproduce a connection reset followed by connector restart and inspect the query and connection configuration involved. Done means the connector restarts without exhausting the 12 GB heap and resumes processing records.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.