alibaba / alibaba/DataX

datax读取hive orc格式的表数据丢失

Open
#267 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

反馈个问题: 在emr hive上的表存储格式是orc格式,使用spark-sql生成的数据用datax导出到mysql存在丢数据现象,之前阿里同学排查出问题是datax的orc相关包版本低不兼容,这个后期有相关的处理嘛?

环境:
hive-conf-2.0.1-1.0.0、spark-conf-2.1.1-1.0.0

场景:
spark-sql> insert overwrite table zr_***_pre select * from zr_***_p where QBM IN ('330401','330402','330411','330421') and tong_time <= '2018-06-05';
...
Time taken: 39.972 seconds
spark-sql> select count(1) from zr_***_pre;
3097270
Time taken: 0.766 seconds, Fetched 1 row(s)

2019-01-24 09:56:15.921 [job-0] INFO JobContainer -
任务启动时刻 : 2019-01-24 09:37:32
任务结束时刻 : 2019-01-24 09:56:15
任务总计耗时 : 1123s
任务平均流量 : 10.85MB/s
记录写入速度 : 2674rec/s
读出记录总数 : 2995047
读写失败总数 : 0

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the path from the Spark SQL-generated ORC Hive table to the DataX export into MySQL, comparing the source count of 3,097,270 with DataX's 2,995,047 records. Inspect the DataX ORC reader dependency version and verify the completed export against the source; done means no records are lost and the compatibility issue is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
mysql, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.