alibaba / alibaba/DataX

用datax从odps同步数据到elasticsearch,出现一直等待的情况

Open
#176 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

情况是这样的,我们es有两个同步任务,第一个任务是写入的两个字段,userid、tags。cleanup=True。
每天早上的时候会先执行这个任务。
第二个任务同步写入,其实是追加数据,但是是三个字段,userid,openid,tags。cleanup=False。
这两个同步任务写入的es是同一个index和indextype。
第一个任务可以正常写入。但是第二个任务在写入的时候一直显示的是wait状态。并没有写入。曾经我怀疑是因为字段不一样所以无法导入。后来用python脚本测试了单条追加写入,确认是可以的。
不知道我描述的是否清晰。这个工作比较着急,所以来这里留言了,请帮助我,谢谢。

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Compare the two DataX synchronization jobs, especially their cleanup settings, field sets, shared index and index type, then inspect where the second job enters the wait state; done means the append task completes and writes its records.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java, python
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.