apache / apache/doris

[Enhancement] Support Paimon 2.0 table writes on master

Open
#66,784 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

I searched existing issues and PRs. #65086 tracks general Paimon write support, and #65868 implemented the Java/JNI write path for `branch-4.1`. This issue focuses on bringing that capability to the current `master` architecture and upgrading the shared runtime to Apache Paimon 2.0.

### Description

Current `master` can read Paimon tables through the new Connector SPI, but it still uses Paimon 1.3.1 and has no complete external-table write path. The implementation merged into `branch-4.1` cannot be applied directly because `master` has moved catalog, transaction, planning, and scan/write contracts into the new connector modules.

The desired behavior is:

- use one Paimon 2.0 runtime version across FE serialization, BE JNI, scanner, and writer;
- support `CREATE TABLE`, `DROP TABLE`, `INSERT INTO`, `INSERT OVERWRITE`, and CTAS against Paimon external catalogs;
- support filesystem, HMS, REST, HDFS, and S3-compatible object storage catalogs;
- support append-only and primary-key tables, partitioned writes, and Paimon bucket assignment;
- preserve Doris statement atomicity with FE-coordinated commit, multi-BE commit aggregation, idempotent retry/reconciliation, and abort cleanup;
- map Doris primitive and complex types, including Paimon 2.0 BLOB plus Doris VARIANT and VECTOR encodings;
- keep read compatibility for existing Paimon tables after the runtime upgrade.

### Proposed solution

Port the proven Java SDK/JNI design from #65868 to the current Connector SPI instead of restoring the old datasource implementation:

```text
ConnectorWritePlanProvider
-> PaimonTableSinkOperator
-> PaimonTableWriter
-> JNI Paimon writer
-> Paimon CommitMessage
-> FE ConnectorTransaction
```

The writer should delegate partition routing, bucket/index assignment, schema defaults, file creation, and commit semantics to the Paimon 2.0 SDK. Doris should own query lifecycle, memory limits, commit-message transport, acknowledgement, retry reconciliation, and cleanup.

BLOB and VECTOR can be represented using Paimon 2.0 physical types plus stable column directives. VARIANT should use a lossless Arrow struct carrying value and metadata binaries so nested and typed values survive round trips.

### Validation

The change should include:

- Java unit tests for Arrow conversion, schema mapping, commit codec, JNI writer, connector write planning, and transaction handling;
- C++ unit tests for VARIANT transport, JNI schema metadata, report budgets, acknowledgement, and cleanup;
- regression coverage for create/drop, insert, overwrite, CTAS atomicity, and BLOB/VARIANT/VECTOR;
- live filesystem/HMS/REST/S3-compatible catalog tests against Paimon 2.0 data.

### Are you willing to submit PR?

Yes. A draft implementation for `master` is being validated and will reference this issue.

### Code of Conduct

I agree to follow the Apache Software Foundation Code of Conduct.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the Java/JNI design from #65868 with the current ConnectorWritePlanProvider, PaimonTableSinkOperator, PaimonTableWriter, and ConnectorTransaction flow. Review the proposed unit and regression coverage, then validate that Paimon 2.0 writes, transaction handling, type conversions, cleanup, and filesystem/HMS/REST/S3-compatible catalog tests meet the stated requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java, sql
Domain
backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.