apache / apache/seatunnel

[Improve][Core] Support close-table events for bounded multi-table jobs

Open
#10,873 1 comment 0 reactions 1 assignee Claimed by @davidzollo View on GitHub
feature
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Search before creating

I checked the current Apache SeaTunnel issues and pull requests on May 11, 2026 and did not find an existing issue/PR that covers this bounded multi-table close-table lifecycle gap end to end.

### Background

In bounded multi-table jobs, SeaTunnel can know that one logical table has been fully consumed before the whole pipeline finishes. However, the current open-source pipeline does not propagate a table-complete signal to downstream multi-table sinks.

As a result, table-scoped sink resources stay alive until the whole task closes, for example:

- per-table sink writers
- table-scoped buffers
- schema caches
- connection / resource-manager entries owned by multi-table sinks

This becomes more visible for sinks that keep table-level state for many tables in one job.

### Current problem

There are two related gaps today:

1. There is no generic `CloseTableEvent` path from bounded sources through engine / transforms into sinks.
2. In parallel jobs, a single "table finished" event is not enough. If one upstream reader emits a close signal before other readers finish delivering their last records for the same table, the sink may reclaim the table too early.

So the missing behavior is not only "emit one event", but "emit a safe per-table completion signal that works with multiple upstream readers".

### Expected behavior

For bounded multi-table jobs:

- participating upstream readers should emit close-table progress after they finish producing records for a table
- downstream transforms should preserve that control event
- multi-table sinks should only close a table after they have received all expected upstream close-table signals for that table

This lets sinks reclaim table-scoped resources early without breaking record ordering in parallel pipelines.

### Proposed direction

Introduce a table lifecycle control path for bounded multi-table jobs:

- add a generic close-table event in `seatunnel-api`
- allow engine source enumerators to send reader events when table-level progress completes
- propagate close-table events through transform and sink flow lifecycles
- let `MultiTableSinkWriter` count upstream close-table signals and close only when all participating readers have reported completion
- wire one concrete bounded multi-table source path first, for example JDBC

### Impact

This improves memory/resource usage for large bounded multi-table jobs and gives sinks a safe hook to release table-scoped resources before the full job ends.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.