alibaba / alibaba/DataX

hive同步到高斯数据库使用datax全量同步

Open
#1,932 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

数据量大约4000w,同步到高斯。双边表结构一致 ,hive全tring,高斯(类似pg)全varchar
目前问题,同步数据到49.45%之后,不再继续同步,也没有报错。将来源数据的4000w limit至 500w可以同步成功,但是经查询datax没有同步数据量限制。
![1695714343705_C9356C2F-5036-44a8-8A11-8EC4C3D03EED](https://github.com/alibaba/DataX/assets/44109605/7a9635e4-39d3-43fc-acf2-3bfff51634be)

望大神指点迷津,不胜感激
Let’s bless each other with happy futures

Contributor guide

No contributing guide indexed for this repository

Research direction

Use the reported DataX synchronization cases as reproduction inputs: compare the 500w run that succeeds with the 4000w run that stops at 49.45%, then inspect the job configuration and available logs. Done means identifying the cause of the silent stall and documenting or implementing a verified fix for the Hive-to-Gauss transfer.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.