alibaba / alibaba/DataX

同步hive到starrocks遗漏数据

Open
#2,275 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

同步hive最新分区到starrocks时,没有报错,但是同步遗漏数据,hive为orc文件,数据可查,没有问题,主键唯一,同时starrocks的也保持一致的主键,字段类型也没有异常,在同步最后一个文件 table/dt=20250226/000030_0 时 提示 min key = null, max key = null 其他00-29序号的文件都能显示正确行数,多次实验同步,hive总量级为30103387,不同channel设置1/3/5/8/10 结果均为 27062535,单独同步该文件也显示max key = null,其他小文件没问题

之前还有测试结果为:
1、完整241列hive表数据只导出前十几列时,导出也异常缺失
2、将完整数据,数据行数减少写入新的hive表,列保持完整,导出starrocks仍然异常缺失
3、将完整数据只导出前170列到新hive表,导出starrocks数据没有丢失行数

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Hive-to-StarRocks sync for partition table/dt=20250226/000030_0 and compare its row count with the Hive total and other files. Start by checking how this ORC file and the 241-column table are handled across different channel settings; done means the partition transfers all expected rows without min key = null or max key = null.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.