alibaba / alibaba/DataX

mysql to mysql同步大表速度限制问题

Open
#2,014 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

因为源库有最大执行时间限制30s,超30秒后SELECT会停止,所以如何才能配置限制SELECT语句的splitPk的单条查询大小,因为源表行数很大,单条SQL查询量都超40W行,查询时间太长会导致`Query execution was interrupted, maximum statement execution time exceeded`错误

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how DataX's MySQL reader uses splitPk to determine the size of each SELECT, then inspect how statement execution limits are handled. The work would be complete when large-table synchronization can constrain each query enough to avoid the reported 30-second MySQL timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.