apache / apache/seatunnel

[Feature] Report table-level lineage to OpenLineage-compatible receivers (Zeta + Flink)

Open
#12,207 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Motivation

Metadata/lineage platforms that speak [OpenLineage](https://openlineage.io/) (Marquez, Gravitino, DataHub, ...) have no way to learn which tables a SeaTunnel job read from and wrote to. This is an umbrella issue for adding opt-in, table-level lineage reporting so a SeaTunnel job can tell such a receiver, in the standard OpenLineage `RunEvent` shape, what it ran and what it touched.

### What this adds

- A START event when a job (or one of its pipelines) begins, a terminal COMPLETE/FAIL/ABORT event when it ends, and an optional throttled RUNNING heartbeat for long-running streaming jobs.
- Table-level datasets only (no column-level lineage), named per the OpenLineage naming spec.
- Output row-count/byte-size statistics on the terminal event where the engine can obtain them.
- Everything here is **off by default** and fully opt-in via a new `openlineage_enabled` job option.

### Scope

- Engines: **Zeta** and the **Flink 1.20 starter**. Spark and the Flink 13/15 starters are explicitly out of scope (the latter two because `JobStatusHook`, the hook this design relies on, is a Flink 1.16+ API).
- Granularity: table-level only. Column-level lineage would require per-transform field-mapping resolution (including SQL lineage parsing for the `Sql` transform) and is a separate, larger effort.

### Design

Two small new modules carry the engine-agnostic pieces, so neither engine's code needs to know about OpenLineage directly:

- `seatunnel-lineage`: configuration resolution (11 new `openlineage_*` job options), the `LineageEvent`/`LineageDataset` model, dataset naming per the OpenLineage spec, and a pluggable `LineageBackend` SPI.
- `seatunnel-lineage-openlineage`: the default backend — OpenLineage 1.29.0 model classes plus a hand-rolled `httpclient`-based transport (neither `openlineage-java`'s own transport nor SeaTunnel's existing shaded Jackson infrastructure could be reused as-is; see PR #12208 for why).

Each engine gets a thin call site on top of that:

- Zeta: `ZetaLineageReporter`, invoked from the job master.
- Flink (1.20 starter only): `LineageJobStatusHook` (JobManager-side, implements Flink's `JobStatusHook`) plus a client-side registration/statistics path, because attached and detached Flink submissions need different terminal-event handling.

### PR plan

Split into 3 PRs, each based on `dev`, to keep each review scope narrow:

1. **PR #12208**: the two new modules, config options, dependency/license bookkeeping. No engine wiring. Implements #12204.
2. **PR #12209**: the Zeta call site. Built on top of #12208's branch, so its diff includes #12208's changes until that one merges — see the PR description for the exact commit range to review. Implements #12205.
3. **PR #12210**: the Flink call site + cluster deployment docs. Same stacking note as #12209. Implements #12206.

### Validation

Beyond unit/integration tests, this was validated end-to-end against a real Gravitino receiver in a sandbox: Zeta jobs writing to both a Paimon table and a Doris table, and the Flink 1.20 starter submitting the same kind of job in both attached and detached mode — each producing the expected OpenLineage events with row counts that were cross-checked directly against the written tables. Details are in each PR's "How was this patch tested" section.

### Backward compatibility

Purely additive and opt-in. No existing configuration option is renamed, removed, or has its default value changed. No existing SPI contract changes. Nothing happens for any job that doesn't set `openlineage_enabled = true`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the three linked PRs and their stated commit ranges, then read the proposed seatunnel-lineage and seatunnel-lineage-openlineage modules before examining ZetaLineageReporter and the Flink 1.20 LineageJobStatusHook. Done means the opt-in configuration, table-level OpenLineage events, engine call sites, statistics handling, and deployment documentation work as described for Zeta and Flink 1.20 without affecting disabled jobs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.