apache / apache/hudi

[BUG] CREATE INDEX IF NOT EXISTS record_index throws "Index already exists" — ignoreIfExists flag is silently dropped

Open
#18,691 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

## Bug Description

**What happened:**
`CREATE INDEX IF NOT EXISTS record_index ON ()` throws
`HoodieMetadataIndexException: Index already exists: record_index` when the index
already exists. The `IF NOT EXISTS` clause has no effect.

**What you expected:**
With `IF NOT EXISTS`, the command should be a no-op when the index already
exists — matching standard SQL semantics and matching the behavior implied by
the `ignoreIfExists: Boolean` field on `CreateIndexCommand`.

**Steps to reproduce:**
1. Create a Hudi COW table with a record-key column (e.g. `uuid`).
2. `CREATE INDEX record_index ON tbl (uuid)` — succeeds.
3. `CREATE INDEX IF NOT EXISTS record_index ON tbl (uuid)` — throws
`HoodieMetadataIndexException: Index already exists: record_index`.

## Root cause

`CreateIndexCommand` in
`hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/IndexCommands.scala`
parses an `ignoreIfExists: Boolean` field from the SQL but never propagates it
to `HoodieSparkIndexClient.create(...)`:

```scala
} else if (indexName.equals(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX)) {
ValidationUtils.checkArgument(...)
new HoodieSparkIndexClient(sparkSession).create(metaClient, indexName,
HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX, columnsMap,
options.asJava, table.properties.asJava)
// ignoreIfExists is dropped here
}
```

`HoodieSparkIndexClient.create` in
`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/HoodieSparkIndexClient.java`
has no `ignoreIfExists` parameter at all, and `createRecordIndex` unconditionally
throws when the index exists:

```java
String fullIndexName = PARTITION_NAME_RECORD_INDEX;
if (indexExists(metaClient, fullIndexName)) {
throw new HoodieMetadataIndexException("Index already exists: " + userIndexName);
}
```

The same gap exists for the column_stats/bloom_filters/secondary-index branches —
none of the `HoodieSparkIndexClient(...).create(...)` call sites in
`CreateIndexCommand.run` pass through `ignoreIfExists`.

## Suggested fix

1. Add an `ignoreIfExists: boolean` parameter to `HoodieSparkIndexClient.create(...)`.
2. Pass it through from every branch in `CreateIndexCommand.run`.
3. In `createRecordIndex` and `createExpressionOrSecondaryIndex`, return early
(instead of throwing) when the index already exists and `ignoreIfExists == true`.

## Notes

- The expression/secondary-index path (`createExpressionOrSecondaryIndex`,
`HoodieSparkIndexClient.java:155-159`) already silently skips re-registration
when the index exists, so observable behavior between record_index and
expression/secondary indexes already differs. The fix is a good opportunity
to unify behavior across both paths.
- `DROP INDEX IF EXISTS` works correctly today: the `ignoreIfNotExists: Boolean`
field on `DropIndexCommand` IS propagated to
`HoodieSparkIndexClient.drop(metaClient, indexName, ignoreIfNotExists)`.
`CREATE INDEX IF NOT EXISTS` is the missing symmetric path.

## Environment

**Hudi version:** 1.x (verified on 1.2; affects all releases where record_index DDL exists)
**Query engine:** Spark 3.3
**Relevant configs:** standard MDT-enabled COW table

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/IndexCommands.scala and trace each create call in CreateIndexCommand.run. Then read HoodieSparkIndexClient.create, createRecordIndex, and createExpressionOrSecondaryIndex in hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/HoodieSparkIndexClient.java. Done means CREATE INDEX IF NOT EXISTS is a no-op for existing indexes across the listed branches, while ordinary CREATE INDEX still reports existing indexes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala, spark
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.