Flink 1.18: support Flink 1.18.1 as a production release line

Objective

Track the build, host-compatibility, state and validation work required to support Apache Flink 1.18.1 on Java 17 as a second production release line, using released Apache Flink 2.2.1 as the reference behavior. Keep 2.2 as the default line with its current coverage and artifact coordinates unchanged.

Prioritize build identity, host API compatibility and state recovery. Connector and format breadth is lower priority because it cannot be validated until the core compiles and installs on the new line.

Evidence and scope

Measured on main 7b5815395703c4b9090f8b85d404ae3cada76286:

• Every released Flink 1.18.1 dependency resolves. No coordinate is missing.
• streamfusion-core reports 99 compiler diagnostics across 21 of 401 Java files.
  They cluster as lookup joins 38, RocksDB integration 26, serializer configuration 16, and a long tail of roughly 20.
• The Arrow boundary, the JNI bridge and the native operator bodies produce zero diagnostics, so the columnar engine itself is not Flink-line coupled.
• The existing loader Java sources compile unchanged against flink-table-api-java:1.18.1.
• Flink 1.18 contains both the FLIP-27 Source API and the planner-loader hook StreamFusion installs through.
• flink:1.18.1-scala_2.12-java17 is published for amd64 and arm64.
• paimon-flink-1.18:2.0.0 targets Flink 1.18.1, which is the Paimon version already pinned.
• flink-connector-kafka:3.2.0-1.18 targets Flink 1.18.0 with Kafka clients 3.4.0 and is the last Kafka connector published for that line.
• Delta publishes no Flink 1.18 build. Delta 3.3.3 targets Flink 1.16.2 and Delta 4.3.1 targets Flink 2.0.1.
• StreamFusion implements Flink's sink2 interfaces directly in two files, both in the Delta sink. Kafka hands encoded records to Flink's own sink, file sinks use Flink's sink, and Paimon uses Paimon's sink APIs.

The diagnostic count is a floor, not a complete inventory. streamfusion-core excludes connector, format, Delta and Paimon sources, so those modules are unmeasured until the profile exists.

The loader compiling unchanged proves source compatibility only. Runtime classloader behavior under a real cluster is proven by #188, not by that result.

SQL and API examples in the child issues are shapes for minimized regressions, not claims that each example has independently been executed on 1.18.

Support contract

• Support exactly Flink 1.18.1 on Java 17. Flink documents its own Java 17 support in 1.18 as experimental, so Java 11 and Flink 1.18.0 stay outside the contract until they have their own build and test matrix.
• Use released dependencies only. Do not fork, patch or locally publish Flink or any connector.
• Keep one source tree with narrow per-line compatibility sources.
• Constructs absent from Flink 1.18, including Variant and delta join, are not applicable rather than fallback coverage. The host cannot plan them, so no decline reason is reachable.
• Shapes Flink 1.18 can plan but StreamFusion cannot preserve must fall back at planning time with an exact reason.
• The intended canonical-savepoint upgrade direction is Flink 1.18.1 to Flink 2.2 at the same StreamFusion release. Real-cluster validation remains pending in #188, so this is not yet a production support guarantee. The reverse direction is outside the intended contract and still requires clear-rejection verification.

P1: build identity and release coordinates

• #183 - Per-line build profile, line-qualified artifact coordinates, and a loader that fails closed on a mixed-line payload. Payload identity shipped in #191; the release line in #198, with the image payload fix in #204 and the logging provider correction in #203.

P1: host API compatibility

• #184 - Core planner and operator compatibility sources for serializer configuration, legacy file sink packages, state TTL hints, session window specs, Variant and delta join. Shipped in #198; the core compiles with zero diagnostics on 1.18.
• #185 - Lookup join adaptation to the 1.18 code generator, with lifecycle, timeout, cancellation and ordered-completion coverage on both lines. Shared lifecycle coverage shipped in #193; the 1.18 adapters and dual-line verification shipped in #198.

P1: state, recovery and upgrade safety

• #186 - RocksDB package move and positional backend factory adaptation, with incremental reuse, aborted-snapshot cleanup, claim and no-claim restore, and rescaling proven on both lines. Shipped in #198 and #196. Two admitted 1.18 constraints: canonical savepoints project through a temporary heap-backed copy, and native keyed operators require StreamFusion's own memory or RocksDB backend because a stock delegate cannot carry the canonical key-group keys.
• #188 - Cross-line canonical savepoint upgrade from 1.18.1 to 2.2 on real clusters, plus mixed-payload rejection at startup. Payload identity shipped in #191 and container startup rejection in #200. Real-cluster upgrades and Podman validation remain required before announcing production support.

P2: connectors and formats

• #187 - Connector and format parity matrix on 1.18, covering Kafka, Paimon and every shipped format, and deciding Delta availability by audit rather than assumption. All nine module suites, Paimon and the upstream format suites run on both lines; Delta was audited in #197 and is unavailable on 1.18. Trimmed to documenting the Kafka 1.18.0-versus-1.18.1 connector boundary.

Test setup and validation

• #189 - Dual-line CI, per-line upstream suite parameterization, dual-line release packaging, and a release-build Nexmark measurement against stock 1.18.1. CI and the upstream suite matrix already run both lines. Trimmed to automated dual-line release publication. The release+mimalloc 1.18 benchmark shipped in #198 and is recorded in docs/benchmarks/flink118-nexmark.md; all ten measured queries are slower than stock (0.28–0.88x).

Existing related work

• #45 is the implementation research for multi-line support. It targets Flink 2.1.3 rather than 1.18 and is stale against current main, but its per-line source root, artifact coordinate and loader profile structure is the validated precedent the build work should mirror. Take its build structure only; its behavior fixes have already reached main by other routes.
• The Flink 2.1.3 port established that a per-line seam of roughly ten methods is sufficient for a neighbouring line, that the loader's covariant-return break is solved without duplicating the class, and that flink-protobuf pins a different protobuf runtime major per line. Flink 1.18 repeats the protobuf split at 3.21.7 and widens the Calcite split to 1.32.0.

Completion criteria

Each child issue links its implementation, states which shapes execute natively versus fall back, updates the owning documentation page in the same change, and closes when its scoped work ships.

The line is announced as supported only when, on Flink 1.18.1 with no regression on 2.2: both lines build from one tree with distinct coordinates and reject mixed payloads at startup; no Flink 2.x artifact appears in the 1.18 dependency graph; the full Java suite and every tagged module suite passes; the unchanged upstream Flink, Kafka and Paimon suites pass with installation and native-execution evidence; checkpoint, savepoint, restore and rescaling pass for every stateful operator; a job upgrades from 1.18.1 to 2.2 through a canonical savepoint at changed parallelism with output identical to a single-version run; container integration passes on both Docker and Podman; a release-build Nexmark comparison against stock 1.18.1 is recorded with both transpose operators in the measured plan; and the documentation matches the tested contract.

Reassess the remaining compile and parity surface after each child ships rather than treating the original diagnostic counts as additive.

Shipped merge checkpoint

PR #198 merged as c90f5a89 and #200 as 89da01cd, each after all 47 checks passed, including both required aggregate gates. The final main tree matches the tested #200 tree. This ships experimental 1.18 compatibility and mixed-payload startup safety; it does not establish the remaining production upgrade or publication guarantees. Per-line corpora and native/fallback/unclassified execution evidence remain documented in docs/upstream-flink-suite.md.

Start with the remaining child issues #188, #187, and #189, then read docs/upstream-flink-suite.md and docs/benchmarks/flink118-nexmark.md. Verify the stated Flink 1.18.1 and 2.2 build, upgrade, connector, CI, container, and benchmark criteria without regressing 2.2. The issue is an umbrella for cross-cutting release work, so a newcomer should select a scoped child issue first.