alibaba / alibaba/DataX

MysqlWriter startWrite时获取连接hung住

Open
#1,537 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

目的:从hbase迁移大量数据到mysql,运行一段时间后speed变成0,先速度降低,大约15min后,变成0.不清楚为啥获取数据库连接会block

使用arthas分析:
datax writer线程12205获取链接hung死。
datax reader线程12206会使用到PrintStream,然后因为12205一直在占用锁,所以PrintStream一直没有释放。所以被12205阻塞
另外一个writer线程12435,因为com.alibaba.datax.plugin.rdbms.util.DBUtil.connect(DBUtil.java:361)方法有加锁,所以也会被12205这个线程阻塞

初步怀疑有脏数据导致回滚,然后卡死,不知道如何解决

[arthas@4252]$ thread 12205
"0-0-16-writer" Id=12205 RUNNABLE (in native)
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at java.io.PrintStream.write(PrintStream.java:480)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
at java.io.PrintStream.write(PrintStream.java:527)
at java.io.PrintStream.print(PrintStream.java:669)
at java.io.PrintStream.println(PrintStream.java:806)
at com.mysql.jdbc.log.StandardLogger.logInternal(StandardLogger.java:318)
at com.mysql.jdbc.log.StandardLogger.logWarn(StandardLogger.java:240)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1177)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2199)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2230)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2025)
at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:778)
at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:47)
at sun.reflect.GeneratedConstructorAccessor12.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at com.alibaba.datax.plugin.rdbms.util.DBUtil.connect(DBUtil.java:392)
at com.alibaba.datax.plugin.rdbms.util.DBUtil.connect(DBUtil.java:384)
at com.alibaba.datax.plugin.rdbms.util.DBUtil.access$000(DBUtil.java:22)
at com.alibaba.datax.plugin.rdbms.util.DBUtil$3.call(DBUtil.java:322)
at com.alibaba.datax.plugin.rdbms.util.DBUtil$3.call(DBUtil.java:319)
at com.alibaba.datax.common.util.RetryUtil$Retry.call(RetryUtil.java:164)
at com.alibaba.datax.common.util.RetryUtil$Retry.doRetry(RetryUtil.java:111)
at com.alibaba.datax.common.util.RetryUtil.executeWithRetry(RetryUtil.java:30)
at com.alibaba.datax.plugin.rdbms.util.DBUtil.getConnection(DBUtil.java:319)
at com.alibaba.datax.plugin.rdbms.util.DBUtil.getConnection(DBUtil.java:303)
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.prepare(CommonRdbmsWriter.java:245)
at com.alibaba.datax.plugin.writer.mysqlwriter.MysqlWriter$Task.prepare(MysqlWriter.java:73)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:50)
at java.lang.Thread.run(Thread.java:748)

[arthas@4252]$ thread 12206
"0-0-16-reader" Id=12206 BLOCKED on java.io.PrintStream@2ab4bc79 owned by "0-0-16-writer" Id=12205
at java.io.PrintStream.write(PrintStream.java:479)
- blocked on java.io.PrintStream@2ab4bc79
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at java.util.logging.StreamHandler.flush(StreamHandler.java:259)
at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:117)
at java.util.logging.Logger.log(Logger.java:738)
at java.util.logging.Logger.doLog(Logger.java:765)
at java.util.logging.Logger.logp(Logger.java:931)
at org.apache.commons.logging.impl.Jdk14Logger.log(Jdk14Logger.java:99)
at org.apache.commons.logging.impl.Jdk14Logger.info(Jdk14Logger.java:194)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.(RecoverableZooKeeper.java:120)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:155)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:175)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.(ZooKeeperWatcher.java:143)
at org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.(ZooKeeperKeepAliveConnection.java:43)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1657)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:104)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:880)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:636)
at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:181)
at com.alibaba.datax.plugin.reader.hbase11xreader.Hbase11xHelper.getHbaseConnection(Hbase11xHelper.java:56)
at com.alibaba.datax.plugin.reader.hbase11xreader.Hbase11xHelper.getTable(Hbase11xHelper.java:68)
at com.alibaba.datax.plugin.reader.hbase11xreader.HbaseAbstractTask.(HbaseAbstractTask.java:35)
at com.alibaba.datax.plugin.reader.hbase11xreader.MultiVersionTask.(MultiVersionTask.java:29)
at com.alibaba.datax.plugin.reader.hbase11xreader.MultiVersionFixedColumnTask.(MultiVersionFixedColumnTask.java:13)
at com.alibaba.datax.plugin.reader.hbase11xreader.Hbase11xReader$Task.init(Hbase11xReader.java:55)
at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run(ReaderRunner.java:45)
at java.lang.Thread.run(Thread.java:748)

[arthas@4252]$ thread 12435
"0-0-25-writer" Id=12435 BLOCKED on java.lang.Class@66099d27 owned by "0-0-16-writer" Id=12205
at com.alibaba.datax.plugin.rdbms.util.DBUtil.connect(DBUtil.java:361)
- blocked on java.lang.Class@66099d27
at com.alibaba.datax.plugin.rdbms.util.DBUtil.access$000(DBUtil.java:22)
at com.alibaba.datax.plugin.rdbms.util.DBUtil$3.call(DBUtil.java:322)
at com.alibaba.datax.plugin.rdbms.util.DBUtil$3.call(DBUtil.java:319)
at com.alibaba.datax.common.util.RetryUtil$Retry.call(RetryUtil.java:164)
at com.alibaba.datax.common.util.RetryUtil$Retry.doRetry(RetryUtil.java:111)
at com.alibaba.datax.common.util.RetryUtil.executeWithRetry(RetryUtil.java:30)
at com.alibaba.datax.plugin.rdbms.util.DBUtil.getConnection(DBUtil.java:319)
at com.alibaba.datax.plugin.rdbms.util.DBUtil.getConnection(DBUtil.java:303)
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.prepare(CommonRdbmsWriter.java:245)
at com.alibaba.datax.plugin.writer.mysqlwriter.MysqlWriter$Task.prepare(MysqlWriter.java:73)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:50)
at java.lang.Thread.run(Thread.java:748)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing DBUtil.java at lines 319, 361, 384, and 392, then trace MysqlWriter.java Task.prepare and the HBase path through Hbase11xHelper.java. Reproduce the HBase-to-MySQL run if possible and compare the reported thread stacks; done should include a confirmed cause and evidence that connection acquisition no longer hangs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.