alibaba / alibaba/DataX

Mysql Reader and Hdfs writer, some records duplicate but port only single record can't reproduce

Open
#305 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

the step is :
1. Using mysql reader and hdfs writer, the number of record from mysql is 258507, and Datax say"读出记录总数:258507"。
2. then I go to see hive table, the number of the record in hive table is 259461, so i guess some records duplicate
3. run in hive sql: "select fskuid,count(*) from tablenamexxxx group by fskuid having count(*)>1", then some records listed: 66 8 (the value of fskuid is 66, and get 8 duplicates)
4. then I go to Datax Json config file, and modify mysql reader'querySql to add "where fskuid=66" and re-run
5. I get only one record.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the DataX JSON configuration, the MySQL reader's querySql, and the Hive SQL used to find duplicate fskuid values. Reproduce the full MySQL-to-HDFS run and compare source and Hive counts, then trace how the MySQL reader and HDFS writer handle records. Done means identifying why the full run duplicates records while the fskuid=66 query returns one record.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, mysql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.