[BUG] CREATE INDEX IF NOT EXISTS record_index throws "Index already exists" — ignoreIfExists flag is silently dropped
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
## Bug Description
**What happened:**
`CREATE INDEX IF NOT EXISTS record_index ON ()` throws
`HoodieMetadataIndexException: Index already exists: record_index` when the index
already exists. The `IF NOT EXISTS` clause has no effect.
**What you expected:**
With `IF NOT EXISTS`, the command should be a no-op when the index already
exists — matching standard SQL semantics and matching the behavior implied by
the `ignoreIfExists: Boolean` field on `CreateIndexCommand`.
**Steps to reproduce:**
1. Create a Hudi COW table with a record-key column (e.g. `uuid`).
2. `CREATE INDEX record_index ON tbl (uuid)` — succeeds.
3. `CREATE INDEX IF NOT EXISTS record_index ON tbl (uuid)` — throws
`HoodieMetadataIndexException: Index already exists: record_index`.
## Root cause
`CreateIndexCommand` in
`hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/IndexCommands.scala`
parses an `ignoreIfExists: Boolean` field from the SQL but never propagates it
to `HoodieSparkIndexClient.create(...)`:
```scala
} else if (indexName.equals(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX)) {
ValidationUtils.checkArgument(...)
new HoodieSparkIndexClient(sparkSession).create(metaClient, indexName,
HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX, columnsMap,
options.asJava, table.properties.asJava)
// ignoreIfExists is dropped here
}
```
`HoodieSparkIndexClient.create` in
`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/HoodieSparkIndexClient.java`
has no `ignoreIfExists` parameter at all, and `createRecordIndex` unconditionally
throws when the index exists:
```java
String fullIndexName = PARTITION_NAME_RECORD_INDEX;
if (indexExists(metaClient, fullIndexName)) {
throw new HoodieMetadataIndexException("Index already exists: " + userIndexName);
}
```
The same gap exists for the column_stats/bloom_filters/secondary-index branches —
none of the `HoodieSparkIndexClient(...).create(...)` call sites in
`CreateIndexCommand.run` pass through `ignoreIfExists`.
## Suggested fix
1. Add an `ignoreIfExists: boolean` parameter to `HoodieSparkIndexClient.create(...)`.
2. Pass it through from every branch in `CreateIndexCommand.run`.
3. In `createRecordIndex` and `createExpressionOrSecondaryIndex`, return early
(instead of throwing) when the index already exists and `ignoreIfExists == true`.
## Notes
- The expression/secondary-index path (`createExpressionOrSecondaryIndex`,
`HoodieSparkIndexClient.java:155-159`) already silently skips re-registration
when the index exists, so observable behavior between record_index and
expression/secondary indexes already differs. The fix is a good opportunity
to unify behavior across both paths.
- `DROP INDEX IF EXISTS` works correctly today: the `ignoreIfNotExists: Boolean`
field on `DropIndexCommand` IS propagated to
`HoodieSparkIndexClient.drop(metaClient, indexName, ignoreIfNotExists)`.
`CREATE INDEX IF NOT EXISTS` is the missing symmetric path.
## Environment
**Hudi version:** 1.x (verified on 1.2; affects all releases where record_index DDL exists)
**Query engine:** Spark 3.3
**Relevant configs:** standard MDT-enabled COW table
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/IndexCommands.scala and trace each create call in CreateIndexCommand.run. Then read HoodieSparkIndexClient.create, createRecordIndex, and createExpressionOrSecondaryIndex in hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/HoodieSparkIndexClient.java. Done means CREATE INDEX IF NOT EXISTS is a no-op for existing indexes across the listed branches, while ordinary CREATE INDEX still reports existing indexes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala, spark
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100