datafusion-contrib / datafusion-contrib/StreamFusion

Flink 1.18: dual-line CI, upstream suite and release validation

Open
#189 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
120
Forks
8
Avg merge
7h 29m
Merged PRs (30d)
76

Description

Current behavior

CI and the upstream suite already run both Flink lines. What remains is the release half: the
release workflow publishes one line. The Flink 1.18 benchmark is already recorded in #198.

Shipped

  • The CI Java matrix runs the full suite on both lines, and uploads per-line portable SQL audit
    outcomes.
  • All nine tagged connector and format module suites run on both lines.
  • Lake module CI runs Paimon Parquet and ORC on both lines; Delta stays on 2.2.
  • Image integration builds the optimized payload and runs the Session-cluster smoke test on both
    lines, including bin/check-artifacts.sh --flink-line.
  • The upstream suite matrix runs runtime, formats, parquet, orc, kafka, paimon and
    state on both lines, with delta excluded on 1.18 and a separate Delta 3.3 / Flink 1.18 host
    audit job. Each run records result totals and separates native execution, expected fallback and
    cases with no execution contract.
  • The loader is tested on both lines with real planner installation.

The optimized release+mimalloc Flink 1.18 Nexmark comparison shipped in #198 and is linked from
docs/benchmarks.md: recorded method and results.
It keeps the unchanged rowwise generator, blackhole sink and both transposes. All ten measured
queries are slower than stock (0.28–0.88x); the existing Kafka headline table is unchanged.

Remaining

  • Publish and validate both artifact sets from the release workflow: separate Maven staging,
    line-qualified archives and image tags, per-line source and Javadoc artifacts, and
    bin/check-artifacts.sh against each line's artifact set. The workflow currently performs one
    deployment and assembles unsuffixed archives.
  • Update docs/releasing.md with the completed automated dual-line publication process. The
    experimental build/package commands and benchmark methodology are already documented.

Publication of the 1.18 line stays blocked on #188 regardless of this issue; passing the CI and
upstream matrix does not establish real-cluster cross-line upgrade support.

Done when

The release workflow publishes and validates both lines' artifact sets. The required release-build
1.18 benchmark and its methodology are already recorded.

Split from #182.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the release workflow and compare its current single deployment and unsuffixed archive assembly with the dual-line requirements in the issue. Trace bin/check-artifacts.sh for validation of each artifact set, then update docs/releasing.md with the completed process. Done means both lines publish separate Maven, archive, image, source, and Javadoc artifacts and each set is validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
ci-cd, documentation, release
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.