alibaba / alibaba/DataX

读取hive数据丢失

Open
#239 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

我们正使用 datax 从 腾讯 TBDS 的 hive 表中读取数据写入 mongodb .
出现了一下情况。

hive表中 5kw数据 量,datax hdfsreader 仅读取其中 3kw 数据完成了导入。
整个过程 datax 日志中 没有错误和异常,显示读取数 仅3kw .从mongodb查询仅3kw数据。

这种状态 半年 180 天 左右存在 40多天数据 丢失。
另外140天正常。

DATAX 部署在 datanode节点,运行时 同时有 hive 作业并行运行。
是否有资源争抢导致数据丢失的问题?
有没有大佬遇到过类似情况?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the DataX hdfsreader job and its reported read count, comparing the Hive source total with the MongoDB import result. Review the run context described here, including concurrent Hive work on the DataNode, and gather logs or a reproducible case showing why 5kw rows become 3kw. Done means identifying the cause and documenting or correcting the missing-row behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.