使用datax将hdfs的数据拉取到mysql,有数据量翻倍问题
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
这是我的配置文件
`{
"job": {
"setting": {
"speed": {
"channel": "1"
}
},
"content": [
{
"reader": {
"name": "hdfsreader",
"parameter": {
"path": "/user/hive/warehouse/ads_acbi.db/ads_refund_m/data_day=${data_day}",
"defaultFS": "hdfs://${big_data}",
"column": [
"*"
],
"fileType": "orc",
"encoding": "UTF-8",
"fieldDelimiter": "\u0001"
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"column": [
"*"
],
"connection": [
{
"jdbcUrl": "jdbc:mysql://${mysql-ip}:${mysql-port}/${mysql-db}",
"table": [
"ads_refund_m_t"
]
}
],
"preSql": [
"delete from ads_refund_m_t"
],
"password": "${mysql-pass}",
"username": "${mysql-user}",
"writeMode": "insert"
}
}
}
]
}
}
`
数据到了mysql数据量有出现翻倍的现象,出现了翻两倍,翻4倍,翻5倍的现象,可以看图片连接,请问是什么原因,怎么解决?

Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the HDFS reader and MySQL writer settings shown in the issue, especially the ORC path, wildcard columns, and preSql delete. Reproduce the transfer while comparing the source row count with the destination count and inspect the five existing comments for diagnostic details. Done means identifying the duplication cause and documenting a verified configuration or fix that produces matching counts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, mysql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100