datafusion-contrib / datafusion-contrib/StreamFusion
Propose a FileFormatFactory priority upstream to drop the Paimon jar-ordering rule
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 120
- Forks
- 8
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 76
Description
FileFormat.fromIdentifier in Paimon takes the first FileFormatFactory on the classpath that claims an identifier (FormatFactoryUtil.discoverFactory). StreamFusion's Parquet factory registers under Paimon's own parquet identifier, so a deployment must place streamfusion-paimon before paimon-flink-* in Flink's lib/ (the 01-streamfusion-paimon.jar naming rule in docs/connectors/paimon.md), and the planner fails closed when the ordering is wrong.
Propose upstream to Paimon a priority (or "override") mechanism on FileFormatFactory, so an accelerator can declare that it replaces the built-in factory for an identifier regardless of classpath order, and adopt it here once released. Until then the ordering rule and the fail-closed check stay (see divergences/31-paimon-sink-via-format-spi.md).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read docs/connectors/paimon.md and divergences/31-paimon-sink-via-format-spi.md first, then inspect the Paimon FileFormatFactory and FormatFactoryUtil behavior described in the issue. Propose a priority or override mechanism upstream, and adopt it here only after a Paimon release provides it; until then, preserve the ordering rule and fail-closed check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100