apache / apache/hudi

[SUPPORT] Does Hudi re-create record level index during an upsert operation?

Open
#12,783 7 comments 0 reactions 0 assignees View on GitHub
area:index area:metadata-table priority:critical
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

I created a hudi table with record level index and performed upsert operation on it. Now, the first time when I performed the upsert operation, it read the record index file, figured out which files needed an update, and wrote the files to S3. The second time when I performed an upsert on the same table, I saw the record index folder being deleted and recreated under the metadata folder. My hudi table is quite large and this re-creation of the entire record level index is too expensive to support during an upsert operation.

**To Reproduce**

Steps to reproduce the behavior:

1. Create hudi table using insert mode and specify record level index
2. Perform an upsert
3. Perform another upsert

**Expected behavior**

I expect any number of upserts after the initial creation of the record level index to just update the index as required and not re-create the whole index.

**Environment Description**

* Hudi version : 0.15.0

* Spark version : 3.4.1

* Hive version :

* Hadoop version : 3.3.6

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : no

**Additional context**

Config I used for upsert operation:

```
hoodie.embed.timeline.server -> false,
hoodie.parquet.small.file.limit -> 1073741824,
hoodie.metadata.record.index.enable -> true,
hoodie.datasource.write.precombine.field -> $timestampField,
hoodie.datasource.write.payload.class -> org.apache.hudi.common.model.OverwriteWithLatestAvroPayload,
hoodie.metadata.index.column.stats.enable -> true,
hoodie.parquet.max.file.size -> 2147483648,
hoodie.metadata.enable -> true,
hoodie.index.type -> RECORD_INDEX,
hoodie.datasource.write.operation -> upsert,
hoodie.parquet.compression.codec -> snappy,
hoodie.datasource.write.recordkey.field -> $recordKeyField,
hoodie.table.name -> $tableName,
hoodie.datasource.write.table.type -> COPY_ON_WRITE,
hoodie.datasource.write.hive_style_partitioning -> true,
hoodie.write.markers.type -> DIRECT,
hoodie.populate.meta.fields -> true,
hoodie.datasource.write.keygenerator.class -> org.apache.hudi.keygen.SimpleKeyGenerator,
hoodie.upsert.shuffle.parallelism -> 10000,
hoodie.datasource.write.partitionpath.field -> $partitionField
```

I do not see any errors but it doesn't make sense that hudi will clear away my index and recreate it.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the two-upsert sequence with Hudi 0.15.0, Spark 3.4.1, S3, and the listed record-index configuration. Inspect the metadata record-index folder after each upsert and determine whether full recreation is expected or indicates a bug. Done means the behavior is explained and, if incorrect, covered by a reproducible test or targeted fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.