datax读取hive orc格式的表数据丢失
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
反馈个问题: 在emr hive上的表存储格式是orc格式,使用spark-sql生成的数据用datax导出到mysql存在丢数据现象,之前阿里同学排查出问题是datax的orc相关包版本低不兼容,这个后期有相关的处理嘛?
环境:
hive-conf-2.0.1-1.0.0、spark-conf-2.1.1-1.0.0
场景:
spark-sql> insert overwrite table zr_***_pre select * from zr_***_p where QBM IN ('330401','330402','330411','330421') and tong_time <= '2018-06-05';
...
Time taken: 39.972 seconds
spark-sql> select count(1) from zr_***_pre;
3097270
Time taken: 0.766 seconds, Fetched 1 row(s)
2019-01-24 09:56:15.921 [job-0] INFO JobContainer -
任务启动时刻 : 2019-01-24 09:37:32
任务结束时刻 : 2019-01-24 09:56:15
任务总计耗时 : 1123s
任务平均流量 : 10.85MB/s
记录写入速度 : 2674rec/s
读出记录总数 : 2995047
读写失败总数 : 0
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the path from the Spark SQL-generated ORC Hive table to the DataX export into MySQL, comparing the source count of 3,097,270 with DataX's 2,995,047 records. Inspect the DataX ORC reader dependency version and verify the completed export against the source; done means no records are lost and the compatibility issue is resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- mysql, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100