[Feature] Paimon Spark 2025 Roadmap
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Motivation
2025 has arrived, and we would like to thank everyone for the contributions in the past! Here we present the 2025 Paimon Spark roadmap, and welcome to take ownership of them or expand upon them!
| Name | Introduction | Link |
|---------------------------|----------------------------------------------------------------------------------------------------------------------------------|-------|
| Variant Type | [feat] Support Variant Type, unlocking support for semi-structured data. | #4471 |
| Optimized Write | [perf] Optimize table writing, including automatic repartitioning and rebalancing data and so on. | |
| Distributed Planning | [perf] Support distributed planning in the scan phase. | #4864 |
| Dataframe Writer V2 | [feat] Integrate Spark's Dataframe Writer V2. | #5531 |
| Liquid Clustering | [perf] Support liquid clustering. | #4815 |
| Isolation Level | [feat] Transaction isolation that supports more levels, like serializable isolation level. | #4616 |
| Support For Spark Connect | [feat] Support [Spark Connect](https://spark.apache.org/docs/latest/spark-connect-overview.html), calling "Paimon Connect". | |
| Default Value | [feat] Support default values for specified fields. | #5754 |
| Constraints | [feat] Support adding constraints to fields, such as not null or other custom constraints. | |
| Partition Stats | [feat] Support partition stats. | |
| Row Lineage | [feat] Support tracking row lineage. | #6022 |
| Identity Column | [feat] When no explicit values are provided during writing, generate unique values for identity column. | |
| Generated Columns | [feat] Support generated columns whose values are automatically generated based on a user-specified function over other columns. | |
| CDC For Non-PK Table | [feat] Support CDC for non-pk table. | |
Contributor guide
No contributing guide indexed for this repository
Research direction
Review the linked issues (#4471, #4864, #5531, #4815, #4616, #5754, and #6022), then narrow the roadmap to one concrete item. No files, tests, or entry points are named, and completion criteria for the listed work are not specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100