alibaba / alibaba/DataX

使用datax将hdfs的数据拉取到mysql,有数据量翻倍问题

Open
#1,507 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

这是我的配置文件

`{
"job": {
"setting": {
"speed": {
"channel": "1"
}
},
"content": [
{
"reader": {
"name": "hdfsreader",
"parameter": {
"path": "/user/hive/warehouse/ads_acbi.db/ads_refund_m/data_day=${data_day}",
"defaultFS": "hdfs://${big_data}",
"column": [
"*"
],
"fileType": "orc",
"encoding": "UTF-8",
"fieldDelimiter": "\u0001"
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"column": [
"*"
],
"connection": [
{
"jdbcUrl": "jdbc:mysql://${mysql-ip}:${mysql-port}/${mysql-db}",
"table": [
"ads_refund_m_t"
]
}
],
"preSql": [
"delete from ads_refund_m_t"
],
"password": "${mysql-pass}",
"username": "${mysql-user}",
"writeMode": "insert"
}
}
}
]
}
}
`

数据到了mysql数据量有出现翻倍的现象,出现了翻两倍,翻4倍,翻5倍的现象,可以看图片连接,请问是什么原因,怎么解决?
![企业微信截图_16630386331207](https://user-images.githubusercontent.com/113405942/189799094-84c6c090-2c25-4fd3-a577-b6eb6ba4bf43.png)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the HDFS reader and MySQL writer settings shown in the issue, especially the ORC path, wildcard columns, and preSql delete. Reproduce the transfer while comparing the source row count with the destination count and inspect the five existing comments for diagnostic details. Done means identifying the duplication cause and documenting a verified configuration or fix that produces matching counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.