Mysql Reader and Hdfs writer, some records duplicate but port only single record can't reproduce
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
the step is :
1. Using mysql reader and hdfs writer, the number of record from mysql is 258507, and Datax say"读出记录总数:258507"。
2. then I go to see hive table, the number of the record in hive table is 259461, so i guess some records duplicate
3. run in hive sql: "select fskuid,count(*) from tablenamexxxx group by fskuid having count(*)>1", then some records listed: 66 8 (the value of fskuid is 66, and get 8 duplicates)
4. then I go to Datax Json config file, and modify mysql reader'querySql to add "where fskuid=66" and re-run
5. I get only one record.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the DataX JSON configuration, the MySQL reader's querySql, and the Hive SQL used to find duplicate fskuid values. Reproduce the full MySQL-to-HDFS run and compare source and Hive counts, then trace how the MySQL reader and HDFS writer handle records. Done means identifying why the full run duplicates records while the fskuid=66 query returns one record.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, mysql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100