[Enhancement] Support Paimon 2.0 table writes on master
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
I searched existing issues and PRs. #65086 tracks general Paimon write support, and #65868 implemented the Java/JNI write path for `branch-4.1`. This issue focuses on bringing that capability to the current `master` architecture and upgrading the shared runtime to Apache Paimon 2.0.
### Description
Current `master` can read Paimon tables through the new Connector SPI, but it still uses Paimon 1.3.1 and has no complete external-table write path. The implementation merged into `branch-4.1` cannot be applied directly because `master` has moved catalog, transaction, planning, and scan/write contracts into the new connector modules.
The desired behavior is:
- use one Paimon 2.0 runtime version across FE serialization, BE JNI, scanner, and writer;
- support `CREATE TABLE`, `DROP TABLE`, `INSERT INTO`, `INSERT OVERWRITE`, and CTAS against Paimon external catalogs;
- support filesystem, HMS, REST, HDFS, and S3-compatible object storage catalogs;
- support append-only and primary-key tables, partitioned writes, and Paimon bucket assignment;
- preserve Doris statement atomicity with FE-coordinated commit, multi-BE commit aggregation, idempotent retry/reconciliation, and abort cleanup;
- map Doris primitive and complex types, including Paimon 2.0 BLOB plus Doris VARIANT and VECTOR encodings;
- keep read compatibility for existing Paimon tables after the runtime upgrade.
### Proposed solution
Port the proven Java SDK/JNI design from #65868 to the current Connector SPI instead of restoring the old datasource implementation:
```text
ConnectorWritePlanProvider
-> PaimonTableSinkOperator
-> PaimonTableWriter
-> JNI Paimon writer
-> Paimon CommitMessage
-> FE ConnectorTransaction
```
The writer should delegate partition routing, bucket/index assignment, schema defaults, file creation, and commit semantics to the Paimon 2.0 SDK. Doris should own query lifecycle, memory limits, commit-message transport, acknowledgement, retry reconciliation, and cleanup.
BLOB and VECTOR can be represented using Paimon 2.0 physical types plus stable column directives. VARIANT should use a lossless Arrow struct carrying value and metadata binaries so nested and typed values survive round trips.
### Validation
The change should include:
- Java unit tests for Arrow conversion, schema mapping, commit codec, JNI writer, connector write planning, and transaction handling;
- C++ unit tests for VARIANT transport, JNI schema metadata, report budgets, acknowledgement, and cleanup;
- regression coverage for create/drop, insert, overwrite, CTAS atomicity, and BLOB/VARIANT/VECTOR;
- live filesystem/HMS/REST/S3-compatible catalog tests against Paimon 2.0 data.
### Are you willing to submit PR?
Yes. A draft implementation for `master` is being validated and will reference this issue.
### Code of Conduct
I agree to follow the Apache Software Foundation Code of Conduct.
Contributor guide
Research direction
Start by comparing the Java/JNI design from #65868 with the current ConnectorWritePlanProvider, PaimonTableSinkOperator, PaimonTableWriter, and ConnectorTransaction flow. Review the proposed unit and regression coverage, then validate that Paimon 2.0 writes, transaction handling, type conversions, cleanup, and filesystem/HMS/REST/S3-compatible catalog tests meet the stated requirements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java, sql
- Domain
- backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100