[Feature] Add `replace` parameter to `create_global_index` procedure
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Motivation
Currently, when a user wants to rebuild or replace an existing global index in Paimon, they must execute two separate operations:
1. `CALL sys.drop_global_index(table => '...', index_column => '...', index_type => '...')`
2. `CALL sys.create_global_index(table => '...', index_column => '...', index_type => '...')`
This two-step process has the following drawbacks:
- **Non-atomic**: Between the drop and create, the table has no index, which may affect concurrent queries relying on the global index.
- **Verbose**: Users must issue two commands for what is logically a single "rebuild" operation.
- **Error-prone**: If the second step fails, the user is left with no index at all.
### Proposal
Add an optional `replace` parameter (default: `false`) to the `create_global_index` procedure in both Spark and Flink. When `replace=true`:
- If an index of the same type on the same column already exists, atomically replace it (delete old index files and commit new index files in a single transaction).
- If no existing index is found, behave as normal create.
When `replace=false` (default):
- If an existing index of the same type on the same column already exists, either fail with an error or proceed with creating a new one (current behavior).
### Reference
Lance (a columnar data format for ML) implements a similar pattern in their `CreateIndexBuilder`:
- https://github.com/lance-format/lance/blob/v7.0.0/rust/lance/src/index/create.rs#L96-L210
Key snippet from Lance:
```rust
pub fn replace(mut self, replace: bool) -> Self {
self.replace = replace;
self
}
// During execute:
let removed_indices = if self.replace {
self.dataset.load_indices().await?
.iter()
.filter(|idx| idx.name == new_idx.name)
.cloned()
.collect()
} else {
vec![]
};
// Then commit with both new_indices and removed_indices atomically
```
### Expected API
**Spark:**
```sql
CALL sys.create_global_index(
table => 'db.my_table',
index_column => 'user_id',
index_type => 'btree',
replace => true
)
```
**Flink:**
```sql
CALL sys.create_global_index(
`table` => 'db.my_table',
index_column => 'user_id',
index_type => 'btree',
`replace` => true
)
```
### Additional Context
Related Paimon source files:
- `paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/CreateGlobalIndexProcedure.java`
- `paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/CreateGlobalIndexProcedure.java`
- `paimon-core/src/main/java/org/apache/paimon/globalindex/btree/BTreeGlobalIndexBuilder.java`
- `paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/globalindex/GenericIndexTopoBuilder.java`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Spark and Flink CreateGlobalIndexProcedure.java files named in the issue, then trace how BTreeGlobalIndexBuilder.java and GenericIndexTopoBuilder.java create and commit index files. Confirm how existing indexes are identified and how both procedures expose parameters. Done means replace=true supports replacement in both engines while the default behavior remains unchanged and replacement commits are atomic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100