同步hive到starrocks遗漏数据
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
同步hive最新分区到starrocks时,没有报错,但是同步遗漏数据,hive为orc文件,数据可查,没有问题,主键唯一,同时starrocks的也保持一致的主键,字段类型也没有异常,在同步最后一个文件 table/dt=20250226/000030_0 时 提示 min key = null, max key = null 其他00-29序号的文件都能显示正确行数,多次实验同步,hive总量级为30103387,不同channel设置1/3/5/8/10 结果均为 27062535,单独同步该文件也显示max key = null,其他小文件没问题
之前还有测试结果为:
1、完整241列hive表数据只导出前十几列时,导出也异常缺失
2、将完整数据,数据行数减少写入新的hive表,列保持完整,导出starrocks仍然异常缺失
3、将完整数据只导出前170列到新hive表,导出starrocks数据没有丢失行数
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the Hive-to-StarRocks sync for partition table/dt=20250226/000030_0 and compare its row count with the Hive total and other files. Start by checking how this ORC file and the 241-column table are handled across different channel settings; done means the partition transfers all expected rows without min key = null or max key = null.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100