mysqlreader big query OOM
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
1.fix 423 row in DBUtil.java
stmt.setFetchSize(fetchSize);
if (stmt instanceof com.mysql.jdbc.Statement) {
((com.mysql.jdbc.Statement)stmt).enableStreamingResults();
}
2. exception:
at com.mysql.jdbc.MysqlIO.nextRowFast([Lcom/mysql/jdbc/Field;IZIZZZ)Lcom/mysql/jdbc/ResultSetRow; (MysqlIO.java:2114)
at com.mysql.jdbc.MysqlIO.nextRow([Lcom/mysql/jdbc/Field;IZIZZZLcom/mysql/jdbc/Buffer;)Lcom/mysql/jdbc/ResultSetRow; (MysqlIO.java:1921)
at com.mysql.jdbc.MysqlIO.readSingleRowSet(JIIZ[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/RowData; (MysqlIO.java:3278)
at com.mysql.jdbc.MysqlIO.getResultSet(Lcom/mysql/jdbc/StatementImpl;JIIIZLjava/lang/String;Z[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetImpl; (MysqlIO.java:462)
at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(Lcom/mysql/jdbc/StatementImpl;IIIZLjava/lang/String;Lcom/mysql/jdbc/Buffer;ZJ[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetImpl; (MysqlIO.java:2997)
at com.mysql.jdbc.MysqlIO.readAllResults(Lcom/mysql/jdbc/StatementImpl;IIIZLjava/lang/String;Lcom/mysql/jdbc/Buffer;ZJ[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetImpl; (MysqlIO.java:2245)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;Ljava/lang/String;Lcom/mysql/jdbc/Buffer;IIIZLjava/lang/String;[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetInternalMethods; (MysqlIO.java:2638)
at com.mysql.jdbc.ConnectionImpl.execSQL(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;ILcom/mysql/jdbc/Buffer;IIZLjava/lang/String;[Lcom/mysql/jdbc/Field;Z)Lcom/mysql/jdbc/ResultSetInternalMethods; (ConnectionImpl.java:2526)
at com.mysql.jdbc.ConnectionImpl.execSQL(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;ILcom/mysql/jdbc/Buffer;IIZLjava/lang/String;[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetInternalMethods; (ConnectionImpl.java:2484)
at com.mysql.jdbc.StatementImpl.executeQuery(Ljava/lang/String;)Ljava/sql/ResultSet; (StatementImpl.java:1446)
at com.alibaba.datax.plugin.rdbms.util.DBUtil.query(Ljava/sql/Statement;Ljava/lang/String;)Ljava/sql/ResultSet; (DBUtil.java:445)
at com.alibaba.datax.plugin.rdbms.util.DBUtil.query(Ljava/sql/Connection;Ljava/lang/String;II)Ljava/sql/ResultSet; (DBUtil.java:431)
at com.alibaba.datax.plugin.rdbms.util.DBUtil.query(Ljava/sql/Connection;Ljava/lang/String;I)Ljava/sql/ResultSet; (DBUtil.java:409)
at com.alibaba.datax.plugin.rdbms.reader.CommonRdbmsReader$Task.startRead(Lcom/alibaba/datax/common/util/Configuration;Lcom/alibaba/datax/common/plugin/RecordSender;Lcom/alibaba/datax/common/plugin/TaskPluginCollector;I)V (CommonRdbmsReader.java:197)
at com.alibaba.datax.plugin.reader.mysqlreader.MysqlReader$Task.startRead(Lcom/alibaba/datax/common/plugin/RecordSender;)V (MysqlReader.java:81)
at com.alibaba.datax.core.taskgroup.runner.ReaderRunner.run()V (ReaderRunner.java:57)
at java.lang.Thread.run()V (Thread.java:722)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in DBUtil.java around the query methods at lines 409-445, then follow the call path from CommonRdbmsReader$Task.startRead to MysqlReader$Task.startRead. Confirm the MySQL reader handles a large query without the reported out-of-memory failure, using the stack trace and proposed fetch configuration as the acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100