datafusion-contrib / datafusion-contrib/StreamFusion
Flink 1.18: connector and format parity matrix
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 120
- Forks
- 8
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 76
Description
Current behavior
Every connector and format StreamFusion ships builds and tests against Flink 1.18.1. What remains
is narrow: the Kafka connector's released 1.18 build targets Flink 1.18.0 rather than 1.18.1, and
that mismatch has not been stated as an admitted support boundary.
Shipped
- Per-line connector and format releases are selected by the
flink-1.18profile:
flink-connector-kafka:3.2.0-1.18withkafka-clients3.4.0,paimon-flink-1.18:2.0.0, the
1.18.1 formats, Calcite 1.32.0 and the protobuf 3.21.7 runtime thatflink-protobuf:1.18.1
generates against. - All nine tagged module suites run on both lines in CI: Kafka, JSON, CSV, Raw, Avro,
Avro-Confluent, Protobuf, Parquet and ORC. - Paimon runs on both lines in CI for Parquet and ORC, and the upstream Paimon suite runs on both
lines against its released 1.18 test contract. - The upstream
formats,parquet,orcandkafkasuites run on both lines. - Delta is decided: #197 audited the released legacy connector against Flink 1.18 and the
Kernel-based implementation now belongs only to the 2.2 source root. There is no admitted
native Delta connector on 1.18, and no 1.18 Delta payload is built or published. This is
unavailable functionality, not a verified host fallback. - Per-line format divergences are enumerated in
docs/flink-compatibility.md: Avro legacy
timestamp mapping only, Parquet map-key nullability in the footer, the 1.18 JSON decoder
rejecting top-level arrays, the absent parser-selection option, and the older Jackson node
factory's decimal spelling carried into JSON, CDC JSON and CSV encoders.
Remaining
- State the Kafka support boundary explicitly.
flink-connector-kafka:3.2.0-1.18is built
against Flink 1.18.0 while StreamFusion targets 1.18.1, and it is the last Kafka connector
published for that line. Record whether that combination is admitted, what was verified, and
that no newer 1.18 connector will arrive.docs/connectors/kafka.mdis the owning page.
Done when
The Kafka connector's 1.18 version boundary is documented as an admitted, verified combination
rather than an implicit one.
Split from #182.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/connectors/kafka.md and compare its Flink 1.18 support statement with the boundary described in docs/flink-compatibility.md. Document the admitted and verified combination of flink-connector-kafka:3.2.0-1.18 with Flink 1.18.1, and state that no newer 1.18 Kafka connector will be published.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 90/100