apache / apache/paimon

[Feature] Add `replace` parameter to `create_global_index` procedure

Open
#8,415 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Motivation

Currently, when a user wants to rebuild or replace an existing global index in Paimon, they must execute two separate operations:

1. `CALL sys.drop_global_index(table => '...', index_column => '...', index_type => '...')`
2. `CALL sys.create_global_index(table => '...', index_column => '...', index_type => '...')`

This two-step process has the following drawbacks:
- **Non-atomic**: Between the drop and create, the table has no index, which may affect concurrent queries relying on the global index.
- **Verbose**: Users must issue two commands for what is logically a single "rebuild" operation.
- **Error-prone**: If the second step fails, the user is left with no index at all.

### Proposal

Add an optional `replace` parameter (default: `false`) to the `create_global_index` procedure in both Spark and Flink. When `replace=true`:
- If an index of the same type on the same column already exists, atomically replace it (delete old index files and commit new index files in a single transaction).
- If no existing index is found, behave as normal create.

When `replace=false` (default):
- If an existing index of the same type on the same column already exists, either fail with an error or proceed with creating a new one (current behavior).

### Reference

Lance (a columnar data format for ML) implements a similar pattern in their `CreateIndexBuilder`:
- https://github.com/lance-format/lance/blob/v7.0.0/rust/lance/src/index/create.rs#L96-L210

Key snippet from Lance:
```rust
pub fn replace(mut self, replace: bool) -> Self {
self.replace = replace;
self
}

// During execute:
let removed_indices = if self.replace {
self.dataset.load_indices().await?
.iter()
.filter(|idx| idx.name == new_idx.name)
.cloned()
.collect()
} else {
vec![]
};
// Then commit with both new_indices and removed_indices atomically
```

### Expected API

**Spark:**
```sql
CALL sys.create_global_index(
table => 'db.my_table',
index_column => 'user_id',
index_type => 'btree',
replace => true
)
```

**Flink:**
```sql
CALL sys.create_global_index(
`table` => 'db.my_table',
index_column => 'user_id',
index_type => 'btree',
`replace` => true
)
```

### Additional Context

Related Paimon source files:
- `paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/CreateGlobalIndexProcedure.java`
- `paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/CreateGlobalIndexProcedure.java`
- `paimon-core/src/main/java/org/apache/paimon/globalindex/btree/BTreeGlobalIndexBuilder.java`
- `paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/globalindex/GenericIndexTopoBuilder.java`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Spark and Flink CreateGlobalIndexProcedure.java files named in the issue, then trace how BTreeGlobalIndexBuilder.java and GenericIndexTopoBuilder.java create and commit index files. Confirm how existing indexes are identified and how both procedures expose parameters. Done means replace=true supports replacement in both engines while the default behavior remains unchanged and replacement commits are atomic.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.