[SUPPORT] Does Hudi re-create record level index during an upsert operation?
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
I created a hudi table with record level index and performed upsert operation on it. Now, the first time when I performed the upsert operation, it read the record index file, figured out which files needed an update, and wrote the files to S3. The second time when I performed an upsert on the same table, I saw the record index folder being deleted and recreated under the metadata folder. My hudi table is quite large and this re-creation of the entire record level index is too expensive to support during an upsert operation.
**To Reproduce**
Steps to reproduce the behavior:
1. Create hudi table using insert mode and specify record level index
2. Perform an upsert
3. Perform another upsert
**Expected behavior**
I expect any number of upserts after the initial creation of the record level index to just update the index as required and not re-create the whole index.
**Environment Description**
* Hudi version : 0.15.0
* Spark version : 3.4.1
* Hive version :
* Hadoop version : 3.3.6
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
Config I used for upsert operation:
```
hoodie.embed.timeline.server -> false,
hoodie.parquet.small.file.limit -> 1073741824,
hoodie.metadata.record.index.enable -> true,
hoodie.datasource.write.precombine.field -> $timestampField,
hoodie.datasource.write.payload.class -> org.apache.hudi.common.model.OverwriteWithLatestAvroPayload,
hoodie.metadata.index.column.stats.enable -> true,
hoodie.parquet.max.file.size -> 2147483648,
hoodie.metadata.enable -> true,
hoodie.index.type -> RECORD_INDEX,
hoodie.datasource.write.operation -> upsert,
hoodie.parquet.compression.codec -> snappy,
hoodie.datasource.write.recordkey.field -> $recordKeyField,
hoodie.table.name -> $tableName,
hoodie.datasource.write.table.type -> COPY_ON_WRITE,
hoodie.datasource.write.hive_style_partitioning -> true,
hoodie.write.markers.type -> DIRECT,
hoodie.populate.meta.fields -> true,
hoodie.datasource.write.keygenerator.class -> org.apache.hudi.keygen.SimpleKeyGenerator,
hoodie.upsert.shuffle.parallelism -> 10000,
hoodie.datasource.write.partitionpath.field -> $partitionField
```
I do not see any errors but it doesn't make sense that hudi will clear away my index and recreate it.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the two-upsert sequence with Hudi 0.15.0, Spark 3.4.1, S3, and the listed record-index configuration. Inspect the metadata record-index folder after each upsert and determine whether full recreation is expected or indicates a bug. Done means the behavior is explained and, if incorrect, covered by a reproducible test or targeted fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100