datafusion-contrib / datafusion-contrib/StreamFusion

Flink 1.18: connector and format parity matrix

Open Beginner friendly
#187 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
120
Forks
8
Avg merge
7h 29m
Merged PRs (30d)
76

Description

Current behavior

Every connector and format StreamFusion ships builds and tests against Flink 1.18.1. What remains
is narrow: the Kafka connector's released 1.18 build targets Flink 1.18.0 rather than 1.18.1, and
that mismatch has not been stated as an admitted support boundary.

Shipped

  • Per-line connector and format releases are selected by the flink-1.18 profile:
    flink-connector-kafka:3.2.0-1.18 with kafka-clients 3.4.0, paimon-flink-1.18:2.0.0, the
    1.18.1 formats, Calcite 1.32.0 and the protobuf 3.21.7 runtime that flink-protobuf:1.18.1
    generates against.
  • All nine tagged module suites run on both lines in CI: Kafka, JSON, CSV, Raw, Avro,
    Avro-Confluent, Protobuf, Parquet and ORC.
  • Paimon runs on both lines in CI for Parquet and ORC, and the upstream Paimon suite runs on both
    lines against its released 1.18 test contract.
  • The upstream formats, parquet, orc and kafka suites run on both lines.
  • Delta is decided: #197 audited the released legacy connector against Flink 1.18 and the
    Kernel-based implementation now belongs only to the 2.2 source root. There is no admitted
    native Delta connector on 1.18, and no 1.18 Delta payload is built or published. This is
    unavailable functionality, not a verified host fallback.
  • Per-line format divergences are enumerated in docs/flink-compatibility.md: Avro legacy
    timestamp mapping only, Parquet map-key nullability in the footer, the 1.18 JSON decoder
    rejecting top-level arrays, the absent parser-selection option, and the older Jackson node
    factory's decimal spelling carried into JSON, CDC JSON and CSV encoders.

Remaining

  • State the Kafka support boundary explicitly. flink-connector-kafka:3.2.0-1.18 is built
    against Flink 1.18.0 while StreamFusion targets 1.18.1, and it is the last Kafka connector
    published for that line. Record whether that combination is admitted, what was verified, and
    that no newer 1.18 connector will arrive. docs/connectors/kafka.md is the owning page.

Done when

The Kafka connector's 1.18 version boundary is documented as an admitted, verified combination
rather than an implicit one.

Split from #182.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/connectors/kafka.md and compare its Flink 1.18 support statement with the boundary described in docs/flink-compatibility.md. Document the admitted and verified combination of flink-connector-kafka:3.2.0-1.18 with Flink 1.18.1, and state that no newer 1.18 Kafka connector will be published.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka
Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
90/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.