apache / apache/seatunnel

A record of the maximum value for a field specified in the source table during data synchronization

Open
#9,999 1 comment 0 reactions 0 assignees View on GitHub
FAQ
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 17h
Merged PRs (30d)
210

Description

How to record the maximum value of a specified source table delta field during distributed execution of tasks using the zeta engine.
For example, I specify a source table field incre_column, want to know the task after successful execution of the maximum value of this field is what, and record to my own business table, so that the next time you can only extract incre_column greater than this value of the record.
To achieve this function, is it possible to modify all source connectors of seatunnel, update the maximum value of the incre_column after each piece of data is read, and call my own interface to record the maximum value after the task is successfully executed. Since the task is multi-node execution, does each node generate a incre_column maximum value? The interface I provide needs to determine which is the real maximum value incre_column and update it according to the task id.
In addition, I see that there is a DynamicCompile transformer in seatunnel. Can I use java code to get the incre_column maximum value of all the data read, and then find a way to pass it to my interface. For example, call the interface directly in DynamicCompile, but since I don't know the jobId in DynamicCompile (can I get it), I can't pass the jobId to the interface, so the interface can't update incre_column maximum value correctly.
Or send an event in DynamicCompile (can it be implemented), pass incre_column maximum value to the event, and finally call my interface in the event handler, pass incre_column maximum value and jobId to the interface.
I am not very familiar with the various mechanisms and code in seatunnel. Is there any good way to implement this function? Is my above idea feasible? If you have any advice, I would be very grateful.

------

在使用zeta引擎进行分布式执行任务的过程中,如何记录一个指定的源表增量字段的最大值。
比如我指定了一个源表的字段incre_column,想知道任务执行成功后本次执行中该字段的最大值是多少,并记录到我自己的业务表里,这样下次执行时就可以只抽取incre_column大于这个值的记录了。
要实现这一功能,是不是可以修改seatunnel的所有source连接器,在其每读取一条数据后,更新incre_column的最大值,等任务执行成功后,调用我自己的接口来记录该最大值。由于任务是多节点执行,是不是每个节点都会产生一个incre_column最大值,我提供的接口里需要根据任务id来判断哪个是incre_column真正的最大值并进行更新。
另外,我看到seatunnel中有一个DynamicCompile的transformer,可以在这个里面使用java代码获取到读取的所有数据的incre_column最大值吗,然后想办法把它传给我的接口。比如直接在DynamicCompile中调用接口,但由于我在DynamicCompile中不知道jobId(可以获取吗),没法把jobId传给接口,所以接口中没办法正确更新incre_column最大值。
或者在DynamicCompile中发送一个事件(可以实现吗),将incre_column最大值传给该事件,最后事件处理器中调用我的接口,将incre_column最大值和jobId传给接口。
我对seatunnel中的各项机制及代码不甚熟悉,请问这个功能有什么好的实现方式,我上述的思路有可行性吗,如有赐教,不胜感激。

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the source connector lifecycle, the DynamicCompile transformer, and the event-handling path to determine where task success, jobId, and distributed aggregation are available. Trace how records and task metadata move across nodes, then define a supported way to persist the maximum incre_column value only after successful execution. Done means the approach is specified with its consistency and multi-node behavior clarified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.