apache / apache/paimon

[Feature] Paimon Spark 2025 Roadmap

Open
#4,816 20 comments 28 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Motivation

2025 has arrived, and we would like to thank everyone for the contributions in the past! Here we present the 2025 Paimon Spark roadmap, and welcome to take ownership of them or expand upon them!

| Name | Introduction | Link |
|---------------------------|----------------------------------------------------------------------------------------------------------------------------------|-------|
| Variant Type | [feat] Support Variant Type, unlocking support for semi-structured data. | #4471 |
| Optimized Write | [perf] Optimize table writing, including automatic repartitioning and rebalancing data and so on. | |
| Distributed Planning | [perf] Support distributed planning in the scan phase. | #4864 |
| Dataframe Writer V2 | [feat] Integrate Spark's Dataframe Writer V2. | #5531 |
| Liquid Clustering | [perf] Support liquid clustering. | #4815 |
| Isolation Level | [feat] Transaction isolation that supports more levels, like serializable isolation level. | #4616 |
| Support For Spark Connect | [feat] Support [Spark Connect](https://spark.apache.org/docs/latest/spark-connect-overview.html), calling "Paimon Connect". | |
| Default Value | [feat] Support default values for specified fields. | #5754 |
| Constraints | [feat] Support adding constraints to fields, such as not null or other custom constraints. | |
| Partition Stats | [feat] Support partition stats. | |
| Row Lineage | [feat] Support tracking row lineage. | #6022 |
| Identity Column | [feat] When no explicit values are provided during writing, generate unique values for identity column. | |
| Generated Columns | [feat] Support generated columns whose values are automatically generated based on a user-specified function over other columns. | |
| CDC For Non-PK Table | [feat] Support CDC for non-pk table. | |

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the linked issues (#4471, #4864, #5531, #4815, #4616, #5754, and #6022), then narrow the roadmap to one concrete item. No files, tests, or entry points are named, and completion criteria for the listed work are not specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.