[Feature][python] Support update-aware incremental refresh for Data Evolution global indexes
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
I found the related work below, but no issue or PR for the Python update-aware refresh path.
### Motivation
In #8407, we gained incremental global-index building across Java/Flink/Spark/Python: repeated builds skip indexed row ranges and cover newly appended data. In #8818, the Java/Core/Flink/Spark path gained DEIX source snapshot metadata and refresh planning for updates within indexed ranges.
I would like to bring that update-aware refresh behavior to PyPaimon for Data Evolution tables.
For example, after building a BTree index on `label`, a user who opts into `global-index.column-update-action=IGNORE` can update `label` within an indexed row range. Python's next `create_global_index` call still treats that range as covered, so it does not rebuild the affected index. Python-built DE indexes also lack the DEIX source snapshot metadata needed by the Java refresh planner.
The default update protection remains in place. This proposal concerns refreshing indexes after users opt into indexed-column updates, not changing that default or promising synchronous index maintenance.
### Solution
I propose following the existing Java behavior in a few reviewable steps:
1. Record Java-compatible DEIX metadata for new Python-built DE indexes, using the snapshot scanned for the build rather than the latest snapshot at commit time.
2. Add update-aware planning in Python: identify affected indexed ranges from the source snapshot and indexed-column changes, and combine them with uncovered ranges.
3. Rebuild the selected ranges and commit old-index removals and new-index additions together, with conflict handling consistent with Java.
For legacy indexes without trustworthy source metadata, retain Java's explicit-rebuild requirement. Keep the current update policy and supported-index restrictions, and account for the write-column metadata changes in #9574.
### Anything else?
I have a local implementation of the first step and have tested DEIX byte compatibility, source snapshot selection, and Python/Java BTree and Bitmap cross-language read/write scenarios. I also checked that Java refresh planners select Python-built indexes after indexed-column updates. I have not implemented the full Python refresh path or validated a complete Spark/Flink refresh job.
@JingsongLi @leaves12138, is anyone already working on the Python side of #8818? Does this scope make sense? I would be happy to contribute, starting with the source metadata PR. If the direction looks good, we can use this issue to track the follow-up work.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at PyPaimon's create_global_index entry point and compare the existing behavior with the Java behavior described in #8818, using #8407 for the incremental-build context. Check the DEIX metadata and source-snapshot scenarios mentioned in the issue, then verify that update-aware planning, range rebuilds, and legacy-index handling meet the stated requirements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100