apache / apache/paimon

[Feature][python] Support update-aware incremental refresh for Data Evolution global indexes

Open
#9,686 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

I found the related work below, but no issue or PR for the Python update-aware refresh path.

### Motivation

In #8407, we gained incremental global-index building across Java/Flink/Spark/Python: repeated builds skip indexed row ranges and cover newly appended data. In #8818, the Java/Core/Flink/Spark path gained DEIX source snapshot metadata and refresh planning for updates within indexed ranges.

I would like to bring that update-aware refresh behavior to PyPaimon for Data Evolution tables.

For example, after building a BTree index on `label`, a user who opts into `global-index.column-update-action=IGNORE` can update `label` within an indexed row range. Python's next `create_global_index` call still treats that range as covered, so it does not rebuild the affected index. Python-built DE indexes also lack the DEIX source snapshot metadata needed by the Java refresh planner.

The default update protection remains in place. This proposal concerns refreshing indexes after users opt into indexed-column updates, not changing that default or promising synchronous index maintenance.

### Solution

I propose following the existing Java behavior in a few reviewable steps:

1. Record Java-compatible DEIX metadata for new Python-built DE indexes, using the snapshot scanned for the build rather than the latest snapshot at commit time.
2. Add update-aware planning in Python: identify affected indexed ranges from the source snapshot and indexed-column changes, and combine them with uncovered ranges.
3. Rebuild the selected ranges and commit old-index removals and new-index additions together, with conflict handling consistent with Java.

For legacy indexes without trustworthy source metadata, retain Java's explicit-rebuild requirement. Keep the current update policy and supported-index restrictions, and account for the write-column metadata changes in #9574.

### Anything else?

I have a local implementation of the first step and have tested DEIX byte compatibility, source snapshot selection, and Python/Java BTree and Bitmap cross-language read/write scenarios. I also checked that Java refresh planners select Python-built indexes after indexed-column updates. I have not implemented the full Python refresh path or validated a complete Spark/Flink refresh job.

@JingsongLi @leaves12138, is anyone already working on the Python side of #8818? Does this scope make sense? I would be happy to contribute, starting with the source metadata PR. If the direction looks good, we can use this issue to track the follow-up work.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at PyPaimon's create_global_index entry point and compare the existing behavior with the Java behavior described in #8818, using #8407 for the incremental-build context. Check the DEIX metadata and source-snapshot scenarios mentioned in the issue, then verify that update-aware planning, range rebuilds, and legacy-index handling meet the stated requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.