[Feature] Report table-level lineage to OpenLineage-compatible receivers (Zeta + Flink)
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 204
Description
### Motivation
Metadata/lineage platforms that speak [OpenLineage](https://openlineage.io/) (Marquez, Gravitino, DataHub, ...) have no way to learn which tables a SeaTunnel job read from and wrote to. This is an umbrella issue for adding opt-in, table-level lineage reporting so a SeaTunnel job can tell such a receiver, in the standard OpenLineage `RunEvent` shape, what it ran and what it touched.
### What this adds
- A START event when a job (or one of its pipelines) begins, a terminal COMPLETE/FAIL/ABORT event when it ends, and an optional throttled RUNNING heartbeat for long-running streaming jobs.
- Table-level datasets only (no column-level lineage), named per the OpenLineage naming spec.
- Output row-count/byte-size statistics on the terminal event where the engine can obtain them.
- Everything here is **off by default** and fully opt-in via a new `openlineage_enabled` job option.
### Scope
- Engines: **Zeta** and the **Flink 1.20 starter**. Spark and the Flink 13/15 starters are explicitly out of scope (the latter two because `JobStatusHook`, the hook this design relies on, is a Flink 1.16+ API).
- Granularity: table-level only. Column-level lineage would require per-transform field-mapping resolution (including SQL lineage parsing for the `Sql` transform) and is a separate, larger effort.
### Design
Two small new modules carry the engine-agnostic pieces, so neither engine's code needs to know about OpenLineage directly:
- `seatunnel-lineage`: configuration resolution (11 new `openlineage_*` job options), the `LineageEvent`/`LineageDataset` model, dataset naming per the OpenLineage spec, and a pluggable `LineageBackend` SPI.
- `seatunnel-lineage-openlineage`: the default backend — OpenLineage 1.29.0 model classes plus a hand-rolled `httpclient`-based transport (neither `openlineage-java`'s own transport nor SeaTunnel's existing shaded Jackson infrastructure could be reused as-is; see PR #12208 for why).
Each engine gets a thin call site on top of that:
- Zeta: `ZetaLineageReporter`, invoked from the job master.
- Flink (1.20 starter only): `LineageJobStatusHook` (JobManager-side, implements Flink's `JobStatusHook`) plus a client-side registration/statistics path, because attached and detached Flink submissions need different terminal-event handling.
### PR plan
Split into 3 PRs, each based on `dev`, to keep each review scope narrow:
1. **PR #12208**: the two new modules, config options, dependency/license bookkeeping. No engine wiring. Implements #12204.
2. **PR #12209**: the Zeta call site. Built on top of #12208's branch, so its diff includes #12208's changes until that one merges — see the PR description for the exact commit range to review. Implements #12205.
3. **PR #12210**: the Flink call site + cluster deployment docs. Same stacking note as #12209. Implements #12206.
### Validation
Beyond unit/integration tests, this was validated end-to-end against a real Gravitino receiver in a sandbox: Zeta jobs writing to both a Paimon table and a Doris table, and the Flink 1.20 starter submitting the same kind of job in both attached and detached mode — each producing the expected OpenLineage events with row counts that were cross-checked directly against the written tables. Details are in each PR's "How was this patch tested" section.
### Backward compatibility
Purely additive and opt-in. No existing configuration option is renamed, removed, or has its default value changed. No existing SPI contract changes. Nothing happens for any job that doesn't set `openlineage_enabled = true`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the three linked PRs and their stated commit ranges, then read the proposed seatunnel-lineage and seatunnel-lineage-openlineage modules before examining ZetaLineageReporter and the Flink 1.20 LineageJobStatusHook. Done means the opt-in configuration, table-level OpenLineage events, engine call sites, statistics handling, and deployment documentation work as described for Zeta and Flink 1.20 without affecting disabled jobs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100