debezium / debezium/dbz

ERROR Failed due to error: Aborting snapshot due to error when last running 'UNLOCK TABLES': Only REPEATABLE READ isolation level is supported for START TRANSACTION WITH CONSISTENT SNAPSHOT in RocksDB Storage Engine. [DBZ-1072]

Open
#196 0 comments 0 reactions 0 assignees View on GitHub
component/mysql-connector migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-1072](https://issues.redhat.com/browse/DBZ-1072)

{code:java}
[2019-01-09 14:12:53,114] INFO rpl_skip_tx_api = OFF (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,114] INFO session_track_gtids = OFF (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO slave_gtid_info = OFF (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO slave_type_conversions = (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO sync_binlog = 1000 (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO system_time_zone = CST (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO time_zone = SYSTEM (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO tls_version = TLSv1,TLSv1.1,TLSv1.2 (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO tokudb_version = 7.5.6 (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO tx_isolation = READ-COMMITTED (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO tx_read_only = OFF (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO version = 5.6.16-log (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO version_comment = Source distribution (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO version_compile_compiler = GNU (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,115] INFO version_compile_compiler_major = 4 (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,116] INFO version_compile_compiler_minor = 8 (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,116] INFO version_compile_machine = x86_64 (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,116] INFO version_compile_os = Linux (io.debezium.connector.mysql.SnapshotReader:801)
[2019-01-09 14:12:53,116] INFO Step 0: disabling autocommit and enabling repeatable read transactions (io.debezium.connector.mysql.SnapshotReader:243)
[2019-01-09 14:12:53,228] INFO Step 1: flush and obtain global read lock to prevent writes to database (io.debezium.connector.mysql.SnapshotReader:265)
[2019-01-09 14:12:53,302] INFO Step 2: start transaction with consistent snapshot (io.debezium.connector.mysql.SnapshotReader:284)
[2019-01-09 14:12:53,379] INFO Step 1: releasing global read lock to enable MySQL writes (io.debezium.connector.mysql.SnapshotReader:658)
[2019-01-09 14:12:53,459] INFO Writes to MySQL tables prevented for a total of 00:00:00.157 (io.debezium.connector.mysql.SnapshotReader:668)
[2019-01-09 14:12:53,460] ERROR Failed due to error: Aborting snapshot due to error when last running 'UNLOCK TABLES': Only REPEATABLE READ isolation level is supported for START TRANSACTION WITH CONSISTENT SNAPSHOT in RocksDB Storage Engine. (io.debezium.connector.mysql.SnapshotReader:187)
org.apache.kafka.connect.errors.ConnectException: Only REPEATABLE READ isolation level is supported for START TRANSACTION WITH CONSISTENT SNAPSHOT in RocksDB Storage Engine. Error code: 3045; SQLSTATE: HY000.
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:208)
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:186)
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:720)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Only REPEATABLE READ isolation level is supported for START TRANSACTION WITH CONSISTENT SNAPSHOT in RocksDB Storage Engine.
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:782)
at com.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:666)
at io.debezium.jdbc.JdbcConnection.executeWithoutCommitting(JdbcConnection.java:1071)
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:286)
... 3 more
[2019-01-09 14:12:53,525] INFO WorkerSourceTask{id=nengyuanyun-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:328)
[2019-01-09 14:12:53,525] INFO WorkerSourceTask{id=nengyuanyun-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:345)
[2019-01-09 14:12:53,526] ERROR WorkerSourceTask{id=nengyuanyun-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:172)
org.apache.kafka.connect.errors.ConnectException: Only REPEATABLE READ isolation level is supported for START TRANSACTION WITH CONSISTENT SNAPSHOT in RocksDB Storage Engine. Error code: 3045; SQLSTATE: HY000.
at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:208)
at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:186)
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:720)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: Only REPEATABLE READ isolation level is supported for START TRANSACTION WITH CONSISTENT SNAPSHOT in RocksDB Storage Engine.
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.StatementImpl.executeInternal(StatementImpl.java:782)
at com.mysql.cj.jdbc.StatementImpl.execute(StatementImpl.java:666)
at io.debezium.jdbc.JdbcConnection.executeWithoutCommitting(JdbcConnection.java:1071)
at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:286)
... 3 more
[2019-01-09 14:12:53,526] ERROR WorkerSourceTask{id=nengyuanyun-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:173)
[2019-01-09 14:12:53,526] INFO [Producer clientId=producer-13] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer:1067)
[2019-01-09 14:13:52,014] INFO WorkerSourceTask{id=nengyuanyun-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:328)
{code}

Contributor guide

Open the contributing guide

Research direction

Start in io.debezium.connector.mysql.SnapshotReader, especially the snapshot execution around lines 243-286 and the failure path near line 720; inspect how the transaction isolation level is set before START TRANSACTION WITH CONSISTENT SNAPSHOT. Trace the JDBC execution through io.debezium.jdbc.JdbcConnection.executeWithoutCommitting. Done means a snapshot using the reported RocksDB configuration completes without the SQLSTATE HY000 error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.